FastISel emits separate load and AND instructions for bitmasking.
(before) %1:i32 = LOAD_I32 %addr; %2:i32 = AND_I32 %1, 255
Fold AND masks into ZExt loads by verifying operands with
maskTrailingOnes. A getFoldedLoadOpcode wrapper is implemented
to manage dispatching logic for better extensibility.
(after) %1:i32 = LOAD8_U_I32 %addr
Fixed: https://github.com/llvm/llvm-project/issues/180783
When folding loads with extensions, the extension operand can be a freeze node
in addition to a load. We can look through it to do the desirability check.
Fixes#184676
This is similar to the FSHL/FSHR handling in
hoistLogicOpWithSameOpcodeHands.
Here the opcodes aren't exactly the same, but the operations are
equivalent.
Fixes regressions from #180888
1. Use ISD::AssertNoFPClass if LoadInst has !nofpclass metadata.
2. Strip ISD::AssertNoFPClass when try to combine load with bitcast
in DAGCombiner::visitBITCAST.
Commit ffd57408efd4 ("[BPF] Enable relocation location for
load/store/shifts") enabled CORE relocation for load/store/shirts. In
particular, the commit did optimization to have load/store/shift insn
itself having the relocation. For the load and store, the optimization
has the following:
rX = *(rY + <relocation>) and *(rX + <relocation>) = rY
There is no value-range check for the above '<relocation>'. For example,
if the original `<relocation>` is 0x10006 due to a large struct, the
insn encoding of `<relocaiton>` will be truncated into '6' and incorrect
result will happen.
This patch fixed the issue by checking the value range of
'<relocation>'. If the `<relocation>` is more than INT16_MAX,
optimization will be skipped.
Even llvm side is fixed, libbpf side may still have issues with the
current approach since libbpf may change the value of <relocation>. If
the <relocation> value is more than INT16_MAX, libbpf will either fail
or need to patch insns.
Let us say we have
rX = *(rY + <relocation>) and *(rX + <relocation>) = rY
libbpf will modify `<relocation>` value depending on the actual offset
in kernel data structure. If `<relocation>` is more than INT16_MAX, more
than one insn will be necessary. llvm can add nop instructions for patch
purpose (see [1]).
The following are major patching cases:
case 1: rX = *(rY + <relocation>) // rX and rY are different
rX = <relocation>
rX += rY
rX = *(rX + 0)
case 2: rX = *(rX + <relocation>)
rX += <relocation>
rX = *(rX + 0)
case 3: *(rX + <relocation>) = imm
rX += <relocation>
*(rX + 0) = imm
rX -= <relocation>
case 4: *(rX + <relocation>) = rY // rX and rY are different
rX += <relocation>
*(rX + 0) = rY
rX -= <relocation>
case 5: *(rX + <relocation>) = rX
// We are not able to resolve this issue.
A llvm option (-disable-bpf-core-optimization) is implemented to
disable bpf core optimizaiton, which means the relocation will not
be in load/store insns. This can workaround the above case 5.
[1] https://github.com/llvm/llvm-project/compare/main...yonghong-song:llvm-project:fix-core-overflow-debug
Adds the WaveActiveBitOr intrinsic from issue #99167. This intrinsic
required a bit more work than the last intrinsics that I have done.
There are some peculiarities, which I verified with dxcompiler:
- WaveActiveBitOr only works on uint and uint64_t, no other types are
allowed
- There is no 16 bit version of WaveActiveBitOr
Followed the checklist:
- [x] Implement WaveActiveBitOr clang builtin,
- [x] Link WaveActiveBitOr clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for WaveActiveBitOr to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for WaveActiveBitOr to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/WaveActiveBitOr.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/WaveActiveBitOr-errors.hlsl
- [x] Create the int_dx_WaveActiveBitOr intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_WaveActiveBitOr to 120 in
DXIL.td
- [x] Create the WaveActiveBitOr.ll and WaveActiveBitOr_errors.ll tests
in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_WaveActiveBitOr intrinsic in IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the WaveActiveBitOr
lowering and map it to int_spv_WaveActiveBitOr in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveBitOr.ll
offload test: https://github.com/llvm/offload-test-suite/pull/487
PR #184882 was missing `half` type-specific overloads for `mul`.
This PR introduces `half` type-specific overloads for `mul` and
additional codegen tests for the half type.
Also added f16 tests for the lowering of llvm.matrix.multiply.
The offload test suite already has a `mul.fp16` test for exercising half
types at runtime, so no change is needed there.
Assisted-by: claude-opus-4.6
This avoids needing to combine the SHL/SHR/OR pattern later.
This improves code quality on RISC-V where our slx/srx instructions
clobber the destination register but we don't have an immediate form.
We can't recover the original direction from the SHL/SHR/OR pattern
and we can't commute it during the TwoAddressInstruction pass like X86
due to the shift amount being in a register.
443ce5569ee9854cfef1139cf6b9cf05165e0902 caused us to start hitting
assertions with non-standard vector widths (<3 x float>) in this case
now that node types are actually enforced. There was a place in
X86ISelLowering.cpp where we just passed along a 64-bit integer whereas
other places constructing a CVTPS2PH node specifically construct a new
integer.
This fixes an assertion I was hitting in the fragment density map sample
in Vulkan Samples. In starfield.frag.hlsl, we have
float starCol = pow((rnd - threshhold) / (1.0 - threshhold), 16.0);
The optimizer recognizes 16.0 as a whole number and converts the call to
`llvm.powi`. The backend goes on to fail with:
fatal error: error in backend: cannot select: %46:fid(s64) = nnan ninf
nsz arcp afn reassoc G_FPOWI %44:fid, %45:iid(s64) (in function:
_Z9starFieldDv3_f)
On Vulkan, there is no integer-exponent for pow. This patch lowers it by
converting the exponent to float and calling GLSL.std.450's Pow.
---------
Co-authored-by: Steven Perron <stevenperron@google.com>
[This
line](af15474262/llvm/lib/Target/AMDGPU/AMDGPUMarkLastScratchLoad.cpp (L121))
in the AMDGPU backend is uncovered by the existing test suite (checked
using coverage, and by asserting that no tests in the existing test
suite fails if we insert an `abort()` at this line).
We propose a test that covers this line. We demonstrate the test by
inserting an `abort()` at that line in commit
[#3cb65cf](3cb65cf445).
Running all tests shows that only our proposed test fails in the
presence of the abort. We'll remove the abort before merging.
This is the only test that fails in the presence of the abort (our new
test) -- it will pass once we remove the abort:
`CodeGen/AMDGPU/mark-last-scratch-load.ll`
The 77bcab835aca1 folds llvm.ptrauth.resign intrinsic in case intrinsic
discriminant and key match those in call ptrauth bundle. However
assertion is now fired in AArch64AsmPrinter when PAC is enabled and
we're tail calling a global, because AUTH_TCRETURN expects address to be
stored in register.
GCC supports the %a modifier with the p constraint (e.g., %a0), while
Clang rejected it. The 'a' modifier means "print as address", which on
a 'p' constraint memory operand is what the default path already does.
Like GCC, reject 'a' with other memory constraints (e.g. 'm').
Close https://github.com/llvm/llvm-project/issues/185343
Patch to add custom lowering for FCANONICALIZE, FMAXNUM_IEEE, and
FMINNUM_IEEE, all of which are required when relying on default
expansion of FMAXIMUMNUM and FMINIMUMNUM.
The lowering is very simple because AArch64's FMAXNM and FMINNM
instructions are IEEE754-2008 compliant, with the implementation
effectively follow the same path take for NEON.
NOTE: Bfloat support will be provided separately.
This is a relatively simple strategy as it is omitting any heuristics for
liveness and register pressure reduction. This works well as the SystemZ ISel
scheduler is using Sched::RegPressure which gives a good input order to begin
with.
It is trying harder with biasing phys regs than GenericScheduler as it also
considers other instructions such as immediate loads directly into phys-regs
produced by the register coalescer. This can hopefully be refactored into
MachineScheduler.cpp.
It has a latency heuristic that is slightly different from the one in
GenericScheduler: It is activated for a specific type of region that have
many "data sequences" consisting of SUs connected only with a single
data-edge that are next to each other in the input order. This is only 3% of
all the scheduling regions, but when activated it is applied on all the
candidates (not just once per cycle). At the same time it is a bit more
careful by checking not only the SU Height against the scheduled latency but
also its Depth against the remaining latency.
It reuses the GenericScheduler handling of weak edges to help copy
coalescing.
It also helps with compare zero elimination as it tries to put a CC-defining
instruction that produces the compare source value above the compare before
any other instruction clobbering CC or the value.
This work was started after observing heavy spilling in Cactus, which was
actually *caused* by GenericScheduler - disabling it (no pre-RA scheduling)
remedied it and gave a 7% improvement in performance on that benchmark. Many
different versions have been tried which has evolved into this initial
simplistic MachineSchedStrategy that does relatively little and yet achieves
double-digit improvements on Cactus and Imagick compared to GenericSched
(which is OTOH 3% better on Blender). There will hopefully be more
improvements added later on as there seems to be potential for it.
It would be very interesting to have other OOO targets try this as well and
perhaps make this available in MachineScheduler.cpp
(A first attempt with improving the pre-RA scheduling was made with #90181,
which however did not materialize in anything actually useful.)
After the change from `report_fatal_error` to `Ctx.emitError` in #142014
there is a necessity to remove unsupported globals. Otherwise there is a
secondary crash during ISel when processing them
Fixes SWDEV-511241
64-bit regpair shift amounts are treated as signed 7-bits, so a
complement
shift amount of 64 (when the primary amount is 0) is sign-extended to
-64,
reversing the shift direction and producing incorrect results. This
affected
any 64-bit rotate or funnel shift where the runtime shift amount could
be 0
(making the complement 64) or >= 64.
Fix by masking the shift amount to [0, 63] and computing the complement
as
(m - 64), which is always in [-64, -1]. Using lsl/lsr (logical shift)
instructions with this negative amount causes the hardware to reverse
the
shift direction while zero-filling vacated positions:
fshl(a, b, amt) = (a << m) | lsl(b, m - 64) // lsl reverses to lsr
fshr(a, b, amt) = (b >> m) | lsr(a, m - 64) // lsr reverses to lsl
where m = amt & 63. The logical shift instructions (lsl/lsr) are used
instead of arithmetic (asl) because asl with a negative amount performs
an
arithmetic right shift that sign-extends, which would corrupt the result
for negative source values.
When m = 0, the complement amount is -64 (magnitude 64), which shifts
all
64 bits out and produces zero, so the complement term vanishes as
required.
[AMDGPU] Fix GFX1250 hazard: S_SET_VGPR_MSB dropped after
S_SETREG_IMM32_B32 (MODE)
On GFX1250, S_SET_VGPR_MSB immediately after S_SETREG_IMM32_B32
targeting
the MODE register is silently dropped by hardware.
AMDGPULowerVGPREncoding may insert S_SET_VGPR_MSB after a setreg(MODE)
in
Case 2 (size > 12) when imm32[12:19] doesn't match current VGPR MSBs, or
when the next VALU instruction needs different MSBs. Fix by inserting
S_NOP
between the setreg and S_SET_VGPR_MSB to prevent the hazard.
The fix handles two scenarios:
- Case 2 mismatch: S_NOP is inserted directly before S_SET_VGPR_MSB in
handleSetregMode.
- Case 2 match followed by a VALU with different MSBs: a flag
(NeedNopBeforeSetVGPRMSB) is set, and setMode inserts S_NOP before the
next S_SET_VGPR_MSB.
Also adds vcmpx-permlane-vgpr-msb-gfx1250.mir to verify that VGPR
lowering
must run after the hazard recognizer: fixVcmpxPermlaneHazards creates
V_MOV_B32 using high VGPRs that need correct S_SET_VGPR_MSB from the
lowering pass.
Without the overload constrain lowering would go with a default path
which would later result in a crash in case if for example AMDGPU asm is
inlined (for example v_ would be Unknown).
Overload sets ConstraintType to be always RegClass for SPIR-V.
The "private global" terminology, likely came from
llvm/lib/IR/Mangler.cpp, is misleading: "private" is the opposite of
"global", and these prefixed symbols are not global in the object file
format sense (e.g. ELF has STB_GLOBAL while these symbols are always
STB_LOCAL). The term "internal symbol" better describes their purpose:
symbols for internal use by compilers and assemblers, not meant to be
visible externally.
This rename is a step toward adopting the "internal symbol prefix"
terminology agreed with GNU as
(https://sourceware.org/pipermail/binutils/2026-March/148448.html).
On RV32 with <N x i64> vectors, inserting a value that is a
sign_extend of an i32 only uses the lower 32 bits, so it can be
lowered without scalar legalization, same as i32 constants.
From issue #99165, adds the implementation of WaveActiveProduct. This
time with the new types for SPIRVTypeInst
- [x] Implement WaveActiveProduct clang builtin,
- [x] Link WaveActiveProduct clang builtin with hlsl_intrinsics.h
- [x] Add sema checks for WaveActiveProduct to
CheckHLSLBuiltinFunctionCall in SemaChecking.cpp
- [x] Add codegen for WaveActiveProduct to EmitHLSLBuiltinExpr in
CGBuiltin.cpp
- [x] Add codegen tests to
clang/test/CodeGenHLSL/builtins/WaveActiveProduct.hlsl
- [x] Add sema tests to
clang/test/SemaHLSL/BuiltIns/WaveActiveProduct-errors.hlsl
- [x] Create the int_dx_WaveActiveProduct intrinsic in
IntrinsicsDirectX.td
- [x] Create the DXILOpMapping of int_dx_WaveActiveProduct to 119 in
DXIL.td
- [x] Create the WaveActiveProduct.ll and WaveActiveProduct_errors.ll
tests in llvm/test/CodeGen/DirectX/
- [x] Create the int_spv_WaveActiveProduct intrinsic in
IntrinsicsSPIRV.td
- [x] In SPIRVInstructionSelector.cpp create the WaveActiveProduct
lowering and map it to int_spv_WaveActiveProduct in
SPIRVInstructionSelector::selectIntrinsic.
- [x] Create SPIR-V backend test case in
llvm/test/CodeGen/SPIRV/hlsl-intrinsics/WaveActiveProduct.ll
Zbkc contains 2 of the 3 instructions from Zbc. Making Zbc imply Zbkc
will make the __riscv_zbkc define be set when Zbc is enabled.
This does not change the diagnostics printed by the assembler.
There's a PR to add this rule to the ISA manual too
https://github.com/riscv/riscv-isa-manual/pull/2524
Fix issue https://github.com/llvm/llvm-project/issues/179100
When lowering the call instruction with illegal type returned, we should
bail out and transfer the lowering to DAG. Otherwise the return value is
not promoted to proper type, but DAG would assume it has been promoted.
---------
Co-authored-by: Yuanke Luo <ykluo@birentech.com>
When the Zilsd extension is enabled on RV32, use SD_RV32/LD_RV32
instructions to spill and restore pairs of callee-saved GPRs instead of
saving 2 separate 32 bit data.
Note that we need to ensure stack slot to be aligned.
Reuse the ExpandIntRes_CLMUL identity to expand vector
CLMUL/CLMULR/CLMULH on wider element types (vXi16, vXi32, vXi64) by
decomposing into half-element-width operations that eventually reach a
legal CLMUL type.
Three generic strategies in expandCLMUL:
1. Halve: halve element width (e.g. v8i16 -> v8i8 on AArch64)
2. promote to double : zext to wider type if CLMUL is legal there (e.g.
x86)
3. Count widen: pad with undef to double element count (e.g. v4i16 ->
v8i16)
A helper canNarrowCLMULToLegal() guides strategy selection and prevents
circular expansion in the CLMULH bitreverse path.
Also add Custom BITREVERSE lowering for v4i16/v8i16 on AArch64 using
REV16+RBIT, which the CLMULH expansion relies on.
Fixes#183768
Existing ISel patterns match setne/seteq following SIMD boolean reductions
any_true and all_true, and drop the ones that are redundant (because the
reductions always return 1 or 0). This adds patterns to also produce eqz
instructions instead of a comparison with a const.
When an Arm64EC function returns a struct by value that is too large for
x64's `RAX` (>8 bytes), the entry thunk synthesizes a hidden sret
pointer parameter for the x64 side. However, this
parameter was never marked with the sret attribute, so ISel did not copy
its value into `x8` (the Arm64EC mapping of `RAX`) on return. This
caused the x64 caller to see a garbage pointer in `RAX` instead of the
return buffer address.
The change adds the sret attribute to the thunk's synthesized pointer
parameter, so that `LowerFormalArguments` saves it and `LowerReturn`
restores it to `x8` before the tail call to `__os_arm64x_dispatch_ret`.
Fixes#185390
\#184194 introduced this test which was failing in some configurations
as it would try and write output to the test directory by having
incorrectly specified -o flags.
### Prefetch Symbol Resolution
Based on this
[suggestion](https://discourse.llvm.org/t/rfc-code-prefetch-insertion/88668/29?u=rlavaee),
we must identify if a prefetch target is defined in the current module
to avoid **undefined symbol errors**. Since this occurs during
sequential **CodeGen**, we must rely on function names rather than IR
Module APIs.
**Key Changes:**
* **`MachineFunction` Integration:** Added a `PrefetchTargets` field
(with serialization) to track all targets associated with a function.
* **Guaranteed Emission:** All prefetch targets are now emitted
regardless of basic block or callsite index matches to ensure the symbol
exists.
* **Fallback Placement:** Targets with non-matching callsite indices are
emitted at the end of the block to resolve the reference.
This commit adds support for lwat/ldat atomic operations with function
code 16 (Compare and Swap Not Equal) via 4 clang builtins:
__builtin_amo_lwat_csne for 32-bit unsigned operations
__builtin_amo_ldat_csne for 64-bit unsigned operations
__builtin_amo_lwat_csne_s for 32-bit signed operations
__builtin_amo_ldat_csne_s for 64-bit signed operations
Function symbols must have a reference to the ADA, because this becomes
the value of the r5 register when the function is called. Simply get the
value from the begin symbol of the section.
