I found that the `SupportedRoundMode` pattern for true16 mode is removed
in https://github.com/llvm/llvm-project/pull/177069 by mistake. Added it
back in this patch and add gfx11 to the test which runs true16 mode
During type legalization, `ReplaceValueWith` handles morphed nodes by
computing `OldValId` and `NewValId`, calling `ReplaceAllUsesOfValueWith`
(RAUW), then adding `ReplacedValues[OldValId] = NewValId`.
However, the RAUW call can trigger `CSE`, which fires NoteDeletion
callbacks that add new entries to `ReplacedValues`. If NoteDeletion adds
`ReplacedValues[A] = B`, and the subsequent assignment adds
`ReplacedValues[B] = A`, a cycle is created. RemapId then recurses
infinitely following the cycle during path compression, causing a stack
overflow crash.
On AArch64, i16 is not a legal type and requires promotion to i32. When
IR uses i16 values as `extractelement` indices that feed into other
`extractelement` operations, the interplay between result promotion and
operand promotion creates the specific CSE pattern that produces the
cycle.
Fix by calling RemapId on both OldValId and NewValId after the RAUW call
to account for new ReplacedValues entries added during the RAUW cascade.
In the reproducer, this causes both ids to resolve to the same value,
skipping the assignment and preventing the cycle.
Fixes https://github.com/llvm/llvm-project/issues/57251
[Assisted-by](https://t.ly/Dkjjk): [Claude Opus
4.6](https://www.anthropic.com/news/claude-opus-4-6)
Add `__arm_atomic_store_with_stshh` implementation as defined in the
ACLE. Validate arguments passed are correct, and lower to the `stshh`
intrinsic plus an atomic store using a pseudo-instruction with the
allowed orderings:
* memory orderings: relaxed, release, seq_cst
* retention policies: keep, strm
The `STSHH` instruction (Store with Store Hint for Hardware) is part
of the `FEAT_PCDPHINT` extension.
Summary
This patch follows on from #178227.
The previous ISel fold lowers the 64-bit case to:
```
fmov d0, x0
fmov d0, d0
```
which is not ideal and could be fmov d0, x0.
A redundant copy comes from the INSERT_SUBREG/INSvi64lane.
This peephole detects <2 x i64> vectors made of a zeroed upper and low
lane produced by FMOVXDr/FMOVDr, then removes the redundant copy.
Further updated tests and added MIR tests.
Use the same VT as the SETcc source, or fall back to using the VT of the
unextended operand of the CTPOP if the element size of the SETcc is too
small to fit the negative popcount.
BPF users expect to see basic block labels (e.g. <L0>, <L1>) in
disassembly output
(https://github.com/llvm/llvm-project/pull/95103#issuecomment-3771234810).
Default --symbolize-operands to on for BPF targets when neither
--symbolize-operands nor --no-symbolize-operands is explicitly
specified.
Add --no-symbolize-operands to allow users to opt out.
minnum/maxnum don't have the correct sNaN semantics, we must convert to
minimumnum/maximumnum instead.
To avoid an NVPTX regression, make it handle fmaximmumnum in one
TableGen pattern.
This is intended as a targeted fix for the miscompile, as the complete
removal of this transform (#93575) appears to be blocked.
Fixes https://github.com/llvm/llvm-project/issues/176624.
In M=4 mode, the behavior matches IEEE 754-2019 minimumNumber, except
that if both operands are sNaN, the result will be sNaN rather than
qNaN. However, this is explicitly allowed for LLVM's minimumnum
intrinsic, as canonicalization can be omitted for non-constrainted FP.
As such, mark fminimumnum/fmaximumnum as legal, and lower them the same
way as fminnum/fmaxnum. In the future, we may wish to switch those to
use M=0 instead, to match IEEE 754-2008 maxNum/minNum instead.
Now that RISCVVLOptimizer has been extended to handle the remaining
cases tryToReduceVL handles, we can remove tryToReduceVL to keep all the
reduction logic in one place.
Intended to be NFC but it looks like in
test/CodeGen/RISCV/rvv/fixed-vectors-insert-subvector-shuffle.ll we were
previously reducing the vl of a volatile load in
insert_subvector_dag_loop, which RISCVVLOptimizer knows to avoid.
On llvm-test-suite and SPEC CPU 2017 -march=rva23u64 -O3 there are no
changes with this patch.
When `handleSetregMode` encounters an `S_SETREG_IMM32_B32` with `Size <=
12`, the instruction's `imm32[12:19]` bits are "free" for VGPR MSB
piggybacking. Previously, the old mode was eagerly written into these
bits, but the `Rewritten` guard in `setMode` would then block subsequent
piggybacking when the next VGPR instruction needed a different mode,
causing an unnecessary `S_SET_VGPR_MSB` to be emitted.
Model the `S_SETREG_IMM32_B32` as the boundary of the old mode's control
range: reset `CurrentMode` and clear `bits[12:19]` to zero. This lets
the next `setMode` call freely piggyback the required mode without
triggering `Rewritten`.
On e.g. aarch64 the TypeSize of scalar types can have a size that is not
known at compile time.
Currently when safestack occurs those it simply crashes as described in
https://github.com/llvm/llvm-project/issues/175868.
Since we cannot verify the size at compile time we simply consider the
access to be unsafe (in regards to safestack).
Reproducer:
```
#include <arm_sve.h>
int main() {
svint32_t vec = svindex_s32(0, 1);
svint32_t res = svadd_s32_z(svptrue_b32(), vec, vec);
int32_t buffer[1024];
svst1_s32(svptrue_b32(), buffer, res);
return 0;
}
```
Initial support for acquire-release atomics, specified as part of
https://github.com/WebAssembly/shared-everything-threads
This adds an ordering operand to atomic loads, stores, RMWs,
wait/notify,
and fences. It currently defaults to 0 and ISel is not updated yet, so
atomics produced by the compiler will still always be seqcst.
Asm parsing and printing, binary emission and disassembly are all
updated. Binary emission will always use the old encoding because the
encoding is smaller, and to get backwards compatibility for free.
RFC
https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544
This patch moves the emission of global variables from
`DwarfDebug::beginModule()` to `DwarfDebug::endModule()`.
It has the following effects:
1. The order of debug entities in the resulting DWARF changes.
2. Currently, if a DISubprogram requires emission of both concrete
out-of-line and inlined subprogram DIEs, and such a subprogram contains
a static local variable, the DIE for the variable is emitted into the
concrete out-of-line subprogram DIE. As a result, the variable is not
available in debugger when breaking at the inlined function instance.
It happens because static locals are emitted in
`DwarfDebug::beginModule()`, but abstract DIEs for functions that are
not completely inlined away are created only later during
`DwarfDebug::endFunctionImpl()` calls.
With this patch, DIEs for static local variables of subprograms that
have both inlined and the concrete out-of-line instances are placed into
abstract subprogram DIEs. They become visible in debugger when breaking
at concrete out-of-line and inlined function instances.
`llvm/test/DebugInfo/Generic/inlined-static-var.ll` illustrates that.
3. It will allow to simplify abstract subprogram DIEs creation by
reverting https://github.com/llvm/llvm-project/pull/159104 later.
This is needed to simplify DWARF emission in a context of proper support
of function-local static variables which comes in the next patch
(https://reviews.llvm.org/D144008), making all function-local entities
handled in `DwarfDebug::endModuleImpl()`.
Authored-by: Kristina Bessonova <kbessonova@accesssoftek.com>
Co-authored-by: David Blaikie <dblaikie@gmail.com>
Co-authored-by: Vladislav Dzhidzhoev <vdzhidzhoev@accesssoftek.com>
We do not update the global registry when expanding a G_GLOBAL_VALUE for
a function pointer. Then during pointer validation, we can get a
garbage value from the global registry.
This PR adds the WaveActiveAllEqual function to HLSL.
It also adds extra macro logic to CGHLSLBuiltins so that you can specify
a different intrinsic name for the SPIRV intrinsic.
Fixes https://github.com/llvm/llvm-project/issues/99162
Some relocations (like R_REF) are emitted to an offset 0 within the CSECT. If other relocations have already been emitted then the relocations are not in increasing order and the linker will emit an error. Sort the relocations before emitting to fix the problem.
This patch adds custom lowering for the following NEON intrinsics to
enable better codegen for convert and load/store operations:
- suqadd
- usqadd
- abs
- sqabs
- sqneg
Closes: https://github.com/llvm/llvm-project/issues/174912
When generating a `mulx` instruction for a widening multiplication, even
if one input is placed in %rdx, LLVM won't place it in the implicit
first slot, instead it'll generate two movs before calling mulx to swap
the registers, which are unnecessary. GCC already has this optimization
(as shown in the issue) so this puts the two compilers closer to each
other on that front.
Co-authored-by: Aiden Grossman <aidengrossman@google.com>
Right now we have a problem where if you have a LLVM module with globals
but no functions, a completely empty SPIR-V module is emitted.
This is because global emission is dependent on tracking intrinsic
functions being emitted in functions.
As a simple fix, just insert a service function, which the backend is
already set up to not actually emit, if there are no real functions.
The current use case of the service function is for function pointers. I
don't think it's possible that we need to both generate a service
function for function pointers and for globals with no functions, so I
just added an error (not an assert) just in case if we do need it for
both cases.
Probably we should rework global handling in the future to work without
these workarounds, but this is a pretty fundamental issue so let's work
around it with this simple change for now.
This change exposed an existing bug:
We consider basic blocks with no successors as fall-through
Also, fix some existing tests. The symptom was:
We previously emitted an empty module, but not that we don't, we hit a
`spirv-val` error about invalid Function StorageClass for globals
because no `addrspace` was specified. Set the `addrspace` to `1`
(`CrossWorkgroup`) in those tests.
Closes: https://github.com/llvm/llvm-project/issues/182899
---------
Signed-off-by: Nick Sarnie <nick.sarnie@intel.com>
Artificial registers were added in
eb0c510ecde667cd911682cc1e855f73f341d134
as a means of giving super-registers heavier weights than that
of their subregisters, even when they only contain a single
physical subregister.
Artifical registers thus do exist in code and participate in
register unit weight calculations, but are not supposed to be
available for register allocation.
This patch completes the support for artificial registers to:
- Ignore artificial registers when joining register unit uber
sets. Artificial registers may be members of classes that
together include registers and their sub-registers, making it
impossible to compute normalised weights for uber sets they
belong to.
We have a use case downstream relying on this being supported,
which allows to avoid introducing a large number of additional
register classes.
- Not generate purely artificial register class intersections.
It is critical not to have such classes, as the common LLVM
codegen infrastructure will try to use them to constrain
classes of virtual registers instead of producing COPYs
whenever both the source and target register classes contain
the same artificial registers.
- Not generate sub-classes where classes with the same
non-artificial members already exist. This is mostly for
convenience. For example, the HI16-capable subset of AMDGPU's
AV_32 is VGPR_32, except VGPR_32 also contains the artificial
staging registers. If the staging registers are not ignored,
we'll end up having an additional generated register class,
AV_32_with_hi16_in_VGPR_16, -- harmless, but also useless.
Eliminates a few inferred AMDGPU register classes:
- VS_32_with_hi16
- VS_32_Lo256_with_hi16
- VS_32_Lo128_with_hi16
- VRegOrLds_32_and_VS_32_Lo256
- VRegOrLds_32_and_VS_32_Lo128
- SRegOrLds_32_and_VRegOrLds_32
Causes no register class changes for other targets.
This breaks a tie where the `SEXT_IN_REG` in an expression like
`SUB(0, SEXT_IN_REG(VECTOR_EXTRACT(..))` can fold into the `SUB` or the
`VECTOR_EXTRACT`. Currently, the `SUB` is always preferred, but it's
better to fold the `SEXT_IN_REG` into the `VECTOR_EXTRACT` extract,
which allows for `SMOV` to be used.
If a function has the same name as a (n OpenCL) builtin, but not
matching number of arguments to the builtin - consider it to be a user
function.
Fixes: https://github.com/llvm/llvm-project/issues/165237
The expansion converts arbitrary-precision FP represented as integer
following these algorithm:
1. Extract sign, exponent, and mantissa bit fields via masks and shifts.
2. Classify the input (zero, denormal, normal, Inf, NaN) using the
exponent and mantissa fields.
3. Normal path: adjusting the exponent bias and left-shifting the
mantissa to fit the wider destination format.
4. Denormal path: normalizing by finding the MSB position of the
mantissa (via count-leading-zeros), computing the correct exponent from
that position, stripping the implicit leading 1, and shifting the
fraction into the destination mantissa field.
5. Assemble the destination IEEE bit pattern (sign | exponent |
mantissa) and select among the normal, denormal, and special-value
results.
Currently only conversions from OCP floats are covered, in LLVM terms
these are: Float8E5M2, Float8E4M3FN, Float6E3M2FN, Float6E2M3FN,
Float4E2M1FN.
OCP spec:
https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
AI has assisted in X86 E2E testing.
When replacing certain AMDGPU library calls with constant data vectors,
the existing implementation only handled single and double-precision
floats. This change extends the functionality to also support
half-precision floats.
Additionally, it refactors the function responsible for generating
constant float data vectors to improve readability and reduces code
duplication. In tandem with this refactoring, the patch relaxes the
check for constant data vectors to include any constant of vector type.
This allows other constant vectors to be processed, such as those
created from constant aggregate zeros (e.g. `<2 x float>
zeroinitializer`).
---------
Signed-off-by: Steffen Holst Larsen <sholstla@amd.com>
The last feature that RISCVVectorPeephole::tryToReduceVL has that
RISCVVLOptimizer doesn't is that if the vl to reduce to doesn't dominate
a vector pseudo, it can sink the pseudo below the vl definition if it's
safe to do so.
This PR shares the logic to check for physical register clobbering in
RISCVInstrInfo, but there may be a better place for it.
The DemandedVLs DenseMap needs to be switched to a MapVector to get
deterministic ordering, since the order in which we sink instructions
can affect the resulting codegen.
This helps removes a few vsetvli toggles in llvm-test-suite.
