For all targets, (V)PADDB is always as fast as (V)PSLLW (usually faster)
- and usually as fast as (V)PAND, and avoids having to load a mask - so
for shift lefts by 2, a pair of (V)PADDB is a better choice vs
(V)PSLLW+(V)PAND
This is only necessary if we're avoiding a (V)PAND mask - otherwise we
just need a single (V)PSLLW.
Extend the existing any_extend(bswap i16) -> rev16 combine to also
handle zero_extend. REV16 preserves a zero upper half, so for i16 loads
this saves one instruction: ldrh+rev+lsr#16 -> ldrh+rev16.
This patch uses DestructiveBinaryShImmUnpred (which was previously
unused as far as I could tell) to define pseudos for arithmetic
immediate instructions such as ADD (immediate), which allows using
MOVPRFX with these instructions.
This patch contains two parts.
1. Update costs reflect to the codegen changes. This is not that
accurate since the step vector can use smaller type if there is a
vscale_range attribute. But we cannot get that in the type-based query
in TTI.
2. Return invalid cost for the vector.extract.last.active that needs
vector split for the step vector. But currently this is not handled
correctly and will hit the assertion.
For not blocking the FindLast reduction in LV
(https://github.com/llvm/llvm-project/pull/184931). We should land this
first and fix the SelectionDAG for vector.extract.last.active lowering.
This is a follow-up of the suggestion left here:
https://github.com/llvm/llvm-project/pull/181707#discussion_r2995733831
The override functions in AMDGPU/ARM/SystemZ/X86 are required to avoid
enabling partial reductions where they were previously disabled (I've
added this for all targets that implement getArithmeticReductionCost).
This addresses an illegal mutation kind, where gisel would hit an
assert. It expands vector elements for non-power2 elements or elements
less that i8 to a power of 2.
A fix to handle vector types correctly was needed in LegalizerHandler.
Fixes#185411
The original attempt (#187051) produced a regression for
`intel-sycl-gpu` because `SPIRVEmitNonSemanticDI` will now self-activate
whenever `llvm.dbg.cu` is present. This removed the need for the
explicit `--spv-emit-nonsemantic-debug-info` flag.
The pass is now entered unconditionally for all SPIR-V targets, but
`NonSemantic.Shader.DebugInfo.100` requires the
`SPV_KHR_non_semantic_info`. Targets like `spirv64-intel` do not enable
that extension by default. When `checkSatisfiable()` ran on those
targets, it issued a fatal error rather than silently skipping.
Adds an early-out from `emitGlobalDI()`: if
`SPV_KHR_non_semantic_info` is not available for the current target, the
pass returns without emitting anything.
There are cases when the `Xqcilsm` vendor extension is enabled that we
are unable to pair non-adjacent load/store instructions. The
`RISCVLoadStoreOptimizer` moves the instruction adjacent to the other
before attempting to pair them but does not move them back when it
fails. This can sometimes prevent the generation of the `Xqcilsm`
load/store multiple instructions. This patch ensures that we move the
unpaired instruction back to it's original location.
We expand vXi8 non-uniform variable rotates as a sequence of uniform
constant rotates along with a SELECT depending on whether the original
rotate amount needs it
This patch removes premature uniform constant rotate expansion to the
OR(SHL,SRL) sequences to allow GFNI targets to use single VGF2P8AFFINEQB
calls
The shadow trimming loop in LowerSTACKMAP hardcoded a 4-byte decrement
per instruction, but when Zca is enabled NOPs are 2 bytes. Use NOPBytes
instead of the hardcoded 4 so the shadow is correctly trimmed on
compressed targets.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When converting a PseudoVMERGE_VVM to PseudoVMV_V_V, we previously
required MIVL <= TrueVL to avoid losing False elements in the tail.
Relax this constraint when the vmerge's False operand equals its
Passthru operand and the True instruction's tail policy is TU
(tail undisturbed). In this case, True's tail lanes preserve its
passthru value (which equals False and Passthru), so the conversion
is safe even when MIVL > TrueVL.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Note the immediates for these 2 instructions in their MachineInstr
representations both use the type width. The SATI_RV64 binary encoding
and the RISCVISD::SATI encoding uses the type width minus one.
Assisted-by: Claude Sonnet 4.5
The `pmin` and `pmax` pattern fragments match `vselect`s, meaning that
the generated operands for `relaxed_min` and `relaxed_max` correspond to
operands 1 and 2 of the `vselect` (rather than operands 0 and 1).
Implements intrinsics used to get the level-of-detail given a texture,
sampler, and a coordinate. It will be used to implement the
corresponding HLSL methods.
Assisted-by: Gemini
Refactor IIT encoding generation. The core change here is that when
generating IIT encodings, we pre-generate all the bits of the IIT
encoding except cases where a type needs to encode its own overload
index, which is patched in later in `TypeInfoGen`. In addition, this
change introduces a class hierarchy for dependent types, so that the
checks in `TypeInfoGen` are more meaningful, and renames/simplifies
several other pieces of code, as listed below.
1. Change the encoding for IIT_ARG's ArgInfo byte to encode the overload
slot index in lower 5 bits and the argument kind in upper 3 bits. This
enabled generating the same packed format for all other dependent types
that need to encode an overload slot index in the IIT encoding. Adjusted
the corresponding C++ code in `IITDescriptor::getArgumentNumber` and
`IIT_Descriptor::getArgumentKind`.
2. Introduce more descriptive classes to handle packing of the overload
index + arg kind into the IIT encoding. `OverloadIndexPlaceholder` is
used to generate a transient value in the type-signature that is patched
in `TypeInfoGen` with that type's overload index. `PackOverloadIndex` is
used to encapsulate the final packing of an overload index and argument
kind in a single byte, and `PatchOverloadIndex` is the class that does
the required patching of a `OverloadIndexPlaceholder` given the type's
overload index.
3. Delete `isAny`, `ArgCode` and `Number` from base `LLVMType` class.
Replace use of `isAny` with `isa<LLVMAnyType>`, `ArgCode` is not used
anymore, and move `Number`, which was used to represent the overload
index for a dependent type to the `LLVMDependentType` class and rename
it to `OverloadIndex`.
4. Introduce `LLVMDependentType` as a base class of all dependent types.
It holds the overload index of the type it depends on in its
`OverloadIndex` field. Also introduce 2 subclasses,
`LLVMFullyDependentType` to represent all fully dependent types (which
encode just the appropriate IIT code and the dependent type's overload
index) and `LLVMPartiallyDependentType` to represent partially dependent
types, that encode the appropriate IIT code and both this type's
overload index and the dependent type's overload index.
5. Change existing dependent type classes to derive from one of these
classes and rename the `num` class argument to `oidx` to better reflect
its meaning.
6. Rename various fields and classes used in `TypeInfoGen` to be more
meaningful. `AssignOverloadIndex` to do overload index assignment,
rename `ACIdxs` to `OverloadIdxs`, `ACTys` to `OverloadTypes` and use
the `DoPatchOverloadIndex` to patch in assigned overload slot indexes.
Remove LowerFCanonicalize. Added fallback for cases when the scalar type also has its Custom lowering to avoid regressions on AMDGPU and SystemZ.
Fixes#143862
When spill slots are eliminated (VGPR-to-AGPR, SGPR-to-VGPR lanes),
debug values referencing these frame indices were not always properly
cleaned up. This caused an assertion failure in getObjectOffset() when
PrologEpilogInserter tried to access the offset of a dead frame object.
The existing debug fixup code in SIFrameLowering and SILowerSGPRSpills
had two limitations:
1. It only checked one operand position, but DBG_VALUE_LIST instructions
can have multiple debug operands with frame indices.
2. It didn't handle all types of dead frame indices uniformly.
Fix by centralizing debug info cleanup in removeDeadFrameIndices(),
which already knows all frame indices being removed. This iterates over
all debug operands using MI.debug_operands().
Assisted-by: Claude Code.
In SIFoldOperands, folding `or x, -1` to `v_mov_b32 -1` removed
`Src1Idx`, which is incorrect because `-1` is in `Src0Idx` (after
canonicalization).
Closes https://github.com/llvm/llvm-project/issues/189677.
vID is a register class for integer vector registers only. vfID (float
vector registers) should be also accepted. Aligning the definition with
other instructions of this kind
related to #188703
Change the gating of `tlbip` instructions (`sysp` aliases) containing
`*E1IS*`, `*E1OS*`, `*E2IS*` or `*E2OS*` to be used with `+tlbid` or
`+d128`. This is because the 2025 Armv9.7-A MemSys specification says:
```
All TLBIP *E1IS*, TLBIP *E1OS*, TLBIP *E2IS* and TLBIP *E2OS*
instructions that are currently dependent on FEAT_D128 are updated
to be dependent on FEAT_D128 or FEAT_TLBID
```
See also change #178912 where the gating of `+d128` for `sysp` was
removed.
Disable generic DAG combines for AMDGPU at -O0 via
disableGenericCombines() to preserve instructions that users may want to
set breakpoints on during debugging.
Assisted-by: Cursor / Claude Opus 4.6
Currently, when the `NoImplicitFloat` attribute is set, consecutive
64-bit load
and store operations will be combined into 128-bit ones, which is
illegal
because that attribute makes floating point and SIMD unavailable.
By implementing the canMergeStoresTo, we can prevent this incorrect
merging.
Fixes https://github.com/llvm/llvm-project/issues/186645
This currently gets stuck in a loop, unable to make progress throught
the
bitcast concat. Limit it to legal element sizes so that we always make
progress
and at least fall back to SDAG as opposed to crashing or getting stuck.
Enable pseudo probes emission for MachO. Due to the 16 character limit
of MachO segment and section, the file sections will be
`__PSEUDO_PROBE,__probes` and `__PSEUDO_PROBE,__probe_descs`.
We have a sshl instruction on RV32 in the 0.21 spec. Unfortunately,
we don't have a SSLLI instruction, but we can put a constant shift
amount in a register.
Replace unsigned with plui or pli_b to better indicate their usage.
Templatize the render function and rename it addSExtImm instead of
addSImm*Unsigned.
This PR enables `G_ADD` and immediate dependencies (relavent ext and
trunc related ops) to be fully legalized and selected.
The most important change made is getting the boilerplate for
RegBankSelect working.
Split from #157161.
This change adds a mechanism to stop emitting `.debug_pubname`,
`.debug_pubtypes` sections for a particular target.
This is particularly useful for cases where IR is generated by frontends
that do not explicitly disable these sections (as `Clang` does for
`NVPTX`), but still use `llc` for code generation.
Currently, only `NVPTX` uses this to disable these sections.
