Rather than creating a dedicated ClangIR test file, the original test file for
this intrinsic is effectively reused:
* clang/test/CodeGen/AArch64/neon-intrinsics.c
“Effectively” meaning that the corresponding test is moved (rather than
literally reused) to a new file within the original AArch64 builtins test
directory:
* clang/test/CodeGen/AArch64/neon/intrinsics.c
This is necessary to avoid lowering unsupported examples from intrinsics.c with
`-fclangir`. The new file will eventually replace the original one once all
builtins from it can be lowered via ClangIR.
To facilitate test re-use, new LIT "feature" is added so that CIR tests can be
run conditionally, e.g. the following will only run when `CLANG_ENABLE_CIR` is
set:
```C
// RUN: %if cir %{%clang_cc1 ... %}
```
This sort of substitutions are documented in [2].
REFERENCES:
[1] https://developer.arm.com/architectures/instruction-sets/intrinsics/#f:@navigationhierarchiessimdisa=[Neon]&q=vceqzd_s64
[2] https://llvm.org/docs/TestingGuide.html#substitutions
Load monitor operations make more sense as atomic operations, as
non-atomic operations cannot be used for inter-thread communication w/o
additional synchronization.
The previous built-in made it work because one could just override the
CPol bits, but that bypasses the memory model and forces the user to learn
about ISA bits encoding.
Making load monitor an atomic operation has a couple of advantages.
First, the memory model foundation for it is stronger. We just lean on the
existing rules for atomic operations. Second, the CPol bits are abstracted away
from the user, which avoids leaking ISA details into the API.
This patch also adds supporting memory model and intrinsics
documentation to AMDGPUUsage.
Solves SWDEV-516398.
Allows __funcref pointers to be used as the element type for WASM tables
in Clang (static, global, zero-length arrays of a reference type).
Modifies `QualType::isWebAssemblyFuncrefType` to correctly look at the
addrspace of the pointee, rather than the pointer type.
Related: #140933
I did not replace riscv.clmulh/clmulr since those require a multiple
instruction pattern match. I wanted to ensure that -O0 will select the
correct instructions without relying on combines.
This change adds the following missing half-precision
add/sub/fma intrinsics for the NVPTX target:
- `llvm.nvvm.add.rn{.ftz}.sat.f16`
- `llvm.nvvm.add.rn{.ftz}.sat.v2f16`
- `llvm.nvvm.mul.rn{.ftz}.sat.f16`
- `llvm.nvvm.mul.rn{.ftz}.sat.v2f16`
- `llvm.nvvm.fma.rn.oob.*`
We lower `fneg` followed by one of the above addition
intrinsics to the corresponding `sub` instruction.
This also removes some incorrect `bf16` fma intrinsics with no
valid lowering.
PTX spec reference:
https://docs.nvidia.com/cuda/parallel-thread-execution/#half-precision-floating-point-instructions
This commit adds 4 Clang builtins for PowerPC AMO store operations:
__builtin_amo_stwat for 32-bit unsigned operations
__builtin_amo_stdat for 64-bit unsigned operations
__builtin_amo_stwat_s for 32-bit signed operations
__builtin_amo_stdat_s for 64-bit signed operations
and maps GCC's AMO store functions to these Clang builtins for
compatibility.
This commit adds 4 Clang builtins for PowerPC AMO load conditional
increment and decrement operations:
__builtin_amo_lwat_cond for 32-bit unsigned operations
__builtin_amo_ldat_cond for 64-bit unsigned operations
__builtin_amo_lwat_cond_s for 32-bit signed operations
__builtin_amo_ldat_cond_s for 64-bit signed operations
This patch adds support in Clang for the RPRFM instruction, by adding
the following intrinsics:
```
void __pldx_range(unsigned int *access_kind*, unsigned int retention_policy,
signed int length*, unsigned int count, signed int stride,
size_t reuse distance, void const *addr);
void __pld_range(unsigned int access_kind*, unsigned int retention_policy,
uint64_t metadata, void const *addr);
```
The `__ARM_PREFETCH_RANGE` macro can be used to test whether these
intrinsics are implemented. If the RPRFM instruction is not available, this
instruction is a NOP.
This implements the following ACLE proposal:
https://github.com/ARM-software/acle/pull/423
Summary:
This is an attempt to begin filling out some missing pieces to allow
more generic compute code to use SPIR-V flavored builtins. This should
provide the basic shuffle operation. The next most important one is the
ballot, but I don't think we have an IR intrinsic for that yet.
I don't know SPIR-V very well so let me know if this is the proper
function with the proper semantic checks.
This commit adds two Clang builtins for AMO load signed operations:
__builtin_amo_lwat_s for 32-bit signed operations
__builtin_amo_ldat_s for 64-bit signed operations
This reverts commit 2c376ffeca490a5732e4fd6e98e5351fcf6d692a because it
breaks assembler.
```
$ llvm-mc -triple=amdgcn -mcpu=gfx1250 -show-encoding <<< "v_wmma_i32_16x16x64_iu8 v[16:23], v[0:7], v[8:15], v[16:23] matrix_b_reuse"
v_wmma_i32_16x16x64_iu8 v[16:23], v[0:7], v[8:15], v[16:23] clamp ; encoding: [0x10,0x80,0x72,0xcc,0x00,0x11,0x42,0x1c]
```
We have a fundamental issue in the clamp support in VOP3P instructions,
which will need more changes.
Fixes#166989
- Adds a clamp immediate operand to the AMDGPU WMMA iu8 intrinsic and
threads it through LLVM IR, MIR lowering, Clang builtins/tests, and MLIR
ROCDL dialect so all layers agree on the new operand
- Updates AMDGPUWmmaIntrinsicModsAB so the clamp attribute is emitted,
teaches VOP3P encoding to accept the immediate, and adjusts Clang
codegen/builtin headers plus MLIR op definitions and tests to match
- Documents what the WMMA clamp operand do
- Implement bitcode AutoUpgrade for source compatibility on WMMA IU8
Intrinsic op
Possible future enhancements:
- infer clamping as an optimization fold based on the use context
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Adding clauses to `CodeGenFunction::EmitX86BuiltinExpr` to convert
SSE4.1/AVX1 builts `roundps/pd/ss/sd` to regular rounding modes.
We use:
1. `roundeven/floor/ceil/trunc` when not using MXCSR or _MM_FROUND_RAISE_EXC , and FP mode is not strict,
2. `experimental_constrained_roundeven/floor/ceil/trunc` when not using MXCSR or _MM_FROUND_RAISE_EXC , and FP mode is strict
3. `x86_sse41/avx_round_ps/pd/ss/sd` when using MXCSR or _MM_FROUND_RAISE_EXC .
Closes#170273
This is the last of the generic instructions created from MVE
intrinsics. It was a little more awkward than the others due to it
taking a Type as one of the arguments. This creates a new function to
create the intrinsic we need.
Implements the ddx and ddy HLSL intrinsics. The DXIL intrinsics call the
ddx_coarse and ddy_builtins as they lower to the coarse variants.
The SPIRV intrinsics lower to their own opcodes OpDPdx and OpDPdy. Tests
are added to ensure the SPIRV builtins are not available outside of
shaders stage.
Closes https://github.com/llvm/llvm-project/issues/99096
Closes https://github.com/llvm/llvm-project/issues/99099
Proposed in [this ACLE
proposal](https://github.com/ARM-software/acle/pull/409), this PR
implements widening FMMLA intrinsics.
- F16 to F32
- MF8 to F32
- MF8 to F16
Additional changes:
- IsOverloadCvt flag renamed to IsOverloadFirstandLast for clarity, as
the name implies conversion. Implementation remains unchanged.
This function is called from various .cpp files under `TargetBuiltins/`,
and was moved unintentionally into `AMDGPU.cpp` in PR #132252. Move it
to a common place.
Let standard casting / builtin_convertvector handle the conversions from BF16 to F32
My only query is how to best implement _mm_cvtpbh_ps - I went for the
v8bf16 -> v8f32 conversion followed by subvector extraction in the end,
but could just as easily extract a v4bf16 first - makes no difference to
final optimized codegen.
First part of #154911
d076608d58d1ec55016eb747a995511e3a3f72aa moved some deps around to avoid
cycles and left clang/Frontend/FrontendDiagnostic.h as a shim that
simply includes clang/Basic/DiagnosticFrontend.h. This PR inlines it so
that nothing in tree still includes clang/Frontend/FrontendDiagnostic.h.
Doing this will help prevent future layering issues. See #162865.
Frontend already depends on Basic, so no new deps need to be added
anywhere except for places that do strict dep checking.
Adds the fwidth intrinsic for HLSL.
The DXIL path only requires modification to the hlsl headers.
The SPIRV path implements the OpFwidth builtin in Clang and instruction
selection for the OpFwidth instruction in LLVM.
Also adds shader stage tests to the ddx_coarse and ddy_coarse
instructions used by fwidth.
Closes#99120
---------
Co-authored-by: Alexander Johnston <alexander.johnston@amd.com>
Now that #152455 is done, we can make all the scalar fma intrinsics to
wrap __builtin_elementwise_fma, which also allows constexpr
The main difference is that FMA4 intrinsics guarantee that the upper
elements are zero, while FMA3 passes through the destination register
elements like older scalar instructions
Fixes#154555
Add support for the following new AArch64 Neon intrinsics:
```
float16x8_t vmmlaq_f16_mf8_fpm(float16x8_t, mfloat8x16_t, mfloat8x16_t, fpm_t);
float32x4_t vmmlaq_f32_mf8_fpm(float32x4_t, mfloat8x16_t, mfloat8x16_t, fpm_t);
```
Per Intel Architecture Instruction Set Extensions Programming Reference
rev. 59 (https://cdrdv2.intel.com/v1/dl/getContent/671368), Revision
History entry for revision -59, AMX-TRANSPOSE was removed
This is the first step in removing some NEON reduction intrinsics that
duplicate the behaviour of their llvm.vector.reduce counterpart.
NOTE: The i8/i16 variants differ in that the NEON versions return an i32
result. However, this looks more about making their code generation
convenient with SelectionDAG disgarding the extra bits. This is only
relevant for the next phase because the Clang usage always truncate
their result, making llvm.vector.reduce a drop in replacement.
`UnqualPtrTy` didn't always match `llvm::PointerType::getUnqual`:
sometimes it returned a pointer that is not in address space 0 (notably
for SPIRV).
Since `UnqualPtrTy` was used as the "generic" or "default" pointer type,
this patch renames it to `DefaultPtrTy` to avoid confusion with LLVM's
`PointerType::getUnqual`.
Define the __dmr2048 type to represent the DMR pair introduced by the
Dense Math Facility on PowerPC, and add three Clang builtins
corresponding to DMF cryptography:
__builtin_mma_dmsha2hash
__builtin_mma_dmsha3hash
__builtin_mma_dmxxshapad
The __dmr2048 type is required for the dmsha3hash crypto builtin, and,
as withother PPC MMA and DMR types, its use is strongly restricted.
AMDGCN flavoured SPIR-V allows AMDGCN specific builtins, including those
for scoped fences and some specific RMWs. However, at present we don't
map syncscopes to their SPIR-V equivalents, but rather use the AMDGCN
ones. This ends up pessimising the resulting code as system scope is
used instead of device (agent) or subgroup (wavefront), so we correct
the behaviour, to ensure that we do the right thing during reverse
translation.
As noted on #156494 and #157403 - its much easier to work with the byte
shift intrinsics if we treat them as vXi8 types instead of vXi64 types
which will require bitcasting
We already do this for the PALIGNR intrinsics which are a more advanced
version of the same shuffle
Builtins for reading the streaming vector length are canonicalised to
use the aarch64.sme.cntsd intrinisic and a multiply, i.e.
- cntsb -> cntsd * 8
- cntsh -> cntsd * 4
- cntsw -> cntsd * 2
This patch also removes the LLVM intrinsics for cnts[b,h,w], and adds
patterns to improve codegen when cntsd is multiplied by a constant.
- Add clang built-ins + sema/codegen
- Add IR Intrinsic + verifier
- Add DAG/GlobalISel codegen for the intrinsics
- Add lowering in SIMemoryLegalizer using a MMO flag.
