Load monitor operations make more sense as atomic operations, as
non-atomic operations cannot be used for inter-thread communication w/o
additional synchronization.
The previous built-in made it work because one could just override the
CPol bits, but that bypasses the memory model and forces the user to learn
about ISA bits encoding.
Making load monitor an atomic operation has a couple of advantages.
First, the memory model foundation for it is stronger. We just lean on the
existing rules for atomic operations. Second, the CPol bits are abstracted away
from the user, which avoids leaking ISA details into the API.
This patch also adds supporting memory model and intrinsics
documentation to AMDGPUUsage.
Solves SWDEV-516398.
This patch implements the `Texture2D` resource type and its `Sample`
member
function in Clang. It includes the necessary AST and Sema changes to
support
the new type and its built-in methods, as well as CodeGen support for
both
DirectX and SPIR-V targets.
Key changes:
- Added `ResourceDimension` to `HLSLAttributedResourceType` and
`HLSLResourceDimension` attribute.
- Implemented `Texture2D` and `SamplerState` in
`HLSLExternalSemaSource`.
- Added `__builtin_hlsl_resource_sample` and associated Sema checking.
- Updated `DirectXTargetCodeGenInfo` and `CommonSPIRTargetCodeGenInfo`
to handle texture types.
- Added AST, Sema, and CodeGen tests for `Texture2D`.
Part 2 of https://github.com/llvm/llvm-project/issues/175630
In the ABIInfo implementations for both the SPIRV and AMDGPU targets,
the lowering of arguments too large to fit into registers is currently
prone to integer overflows when determining the number of needed
registers for the arguments. This causes arguments so large that they
need more registers than an `unsigned` can represent to look like they
fit into the available registers. To avoid this, the function for
determining the required number of registers is changed to return a
64-bit unsigned integer value instead.
Note that the SPIR-V target currently trips the verifier due to a check
that arguments passed by value don't exceed the representable size. This
also affects other targets, such as x86 and is outside the scope of
these changes.
See https://github.com/llvm/llvm-project/issues/118207.
---------
Signed-off-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
Co-authored-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
Summary:
We support variadic functions in AMDGPU / NVPTX via an LLVM-IR pass.
This patch applies the same handling here to support them on this
target.
I am unsure what the ABI should look like here, I have mostly copied the
one we use for NVPTX where it's basically a struct layout with natural
alignment. This wastes some space, which is why AMDGPU does not pad
them.
Additionally, this required allowing the SPIRV_FUNC calling convention.
I'm assuming this is compatible with the C calling convention in IR, but
I will need someone to confirm that for me.
This avoids implicit truncation assertions with
https://github.com/llvm/llvm-project/pull/171456. Apparently the
alignment code path was previously untested.
The first commit switches the test to generated check lines.
Reference issue: https://github.com/ROCm/llvm-project/issues/67
This patch adds support for expanding s_waitcnt instructions into
sequences with decreasing counter values, enabling PC-sampling profilers
to identify which specific memory operation is causing a stall.
This is controlled via:
Clang flag: -mamdgpu-expand-waitcnt-profiling /
-mno-amdgpu-expand-waitcnt-profiling
Function attribute: "amdgpu-expand-waitcnt-profiling"
When enabled, instead of emitting a single waitcnt, the pass generates a
sequence that waits for each outstanding operation individually. For
example, if there are 5 outstanding memory operations and the target is
to wait until 2 remain:
**Original**:
s_waitcnt vmcnt(2)
**Expanded**:
s_waitcnt vmcnt(4)
s_waitcnt vmcnt(3)
s_waitcnt vmcnt(2)
The expansion starts from (Outstanding - 1) down to the target value,
since waitcnt(Outstanding) would be a no-op (the counter is already at
that value).
- Uses ScoreBrackets to determine the actual number of outstanding
operations
- Only expands when operations complete in-order
- Skips expansion for mixed event types (e.g., LDS+SMEM on same counter)
- Skips expansion for scalar memory (always out-of-order)
Releated previous work for Reference
- **PR**: llvm/llvm-project#79236 (related `-amdgpu-waitcnt-forcezero`)
---------
Co-authored-by: Pierre van Houtryve <pierre.vanhoutryve@amd.com>
Implements https://github.com/ARM-software/acle/pull/404
This allows the user to specify "featA+featB;priority=[1-255]" where
priority=255 means highest priority. If the explicit priority string is
omitted then the priority of "featA+featB" is implied, which is lower
than priority=1.
Internally this gets expanded using special FMV features P0 ... P7 which
can encode up to 256-1 priority levels (excluding all zeros). Those do
not have corresponding detection bit at pos FEAT_#enum so I made this
field optional in FMVInfo. Also they don't affect the codegen or name
mangling of versioned functions.
At the moment AMDGCN flavoured SPIRV uses the SPIRV ABI with some tweaks
revolving around passing aggregates as direct. This is problematic in
multiple ways:
- it leads to divergence from code compiled for a concrete target, which
makes it difficult to debug;
- it incurs a run time cost, when dealing with larger aggregates;
- it incurs a compile time cost, when dealing with larger aggregates.
This patch switches over AMDGCN flavoured SPIRV to implement the AMDGPU
ABI (except for dealing with variadic functions, which will be added in
the future). One additional complication (and the primary motivation
behind the current less than ideal state of affairs) stems from `byref`,
which AMDGPU uses, not being expressible in SPIR-V. We deal with this by
CodeGen-ing for `byref`, lowering it to the `FuncParamAttr ByVal` in
SPIR-V, and restoring it when doing reverse translation from AMDGCN
flavoured SPIR-V.
Fixes#154772
We previously set `ptx_kernel` for all kernels. But it's incorrect to
add `ptx_kernel` to the stub version of kernel introduced in #115821.
This patch copies the workaround of AMDGPU.
The clang side of the calling convention code for arm64 vs. arm64ec is
close enough that this isn't really noticeable in most cases, but the
rule for choosing whether to pass a struct directly or indirectly is
significantly different.
(Adapted from my old patch https://reviews.llvm.org/D125419 .)
Fixes#89615.
This change drops the use of the "Layout" type and instead uses explicit
padding throughout the compiler to represent types in HLSL buffers.
There are a few parts to this, though it's difficult to split them up as
they're very interdependent:
1. Refactor HLSLBufferLayoutBuilder to allow us to calculate the padding
of arbitrary types.
2. Teach Clang CodeGen to use HLSL specific paths for cbuffers when
generating aggregate copies, array accesses, and structure accesses.
3. Simplify DXILCBufferAccesses such that it directly replaces accesses
with dx.resource.getpointer rather than recalculating the layout.
4. Basic infrastructure for SPIR-V handling, but the implementation
itself will need work in follow ups.
Fixes several issues, including #138996, #144573, and #156084.
Resolves#147352.
Rather than using a nullable SmallVector, use a wrapper class for offset
info. This simplifies places that need to handle whether or not there's
any offset information.
According to the
[spec](https://github.com/intel/llvm/blob/sycl/sycl/doc/design/spirv-extensions/SPV_INTEL_function_pointers.asciidoc),
it is illegal to addrspacecast to the generic AS, so use the function
pointer AS for null constants.
"It is illegal to use Function Pointer as 'Pointer' argument of
OpPtrCastToGeneric."
This was found when compiling the OpenMP Device RTL for SPIR-V.
Signed-off-by: Nick Sarnie <nick.sarnie@intel.com>
The original [change](https://github.com/llvm/llvm-project/pull/137882)
unifying the device kernel attributes had some inexplicable behavior,
such as `amdgpu_kernel` resulting in a function ending up with the
`spir_kernel` CC but `nvptx_kernel` not doing the same, both cases
compiling for SPIR. There was also a
[crash](https://github.com/llvm/llvm-project/issues/161077).
`sycl_kernel` is now separated out from `device_kernel`, but still there
was some weird behavior for the remaining spellings.
For the target-specific spellings (`nvptx_kernel` and `amdgpu_kernel`),
while not technically required, we warn and ignore the attribute if the
spelling doesn't match the target because it's weird from the user's
point of view to allow it.
Also we make sure that any valid usage actually applies the CC to the
generated `llvm:Function`. This worked for `NVPTX` already but was
missing for `SPIR/SPIR-V` and `AMDGPU`, it needs to be explicitly done
in `TargetInfo`. This allows us to remove the `amdgpu_kernel` specific
handing we had. That special handling was previously required because it
was the only variation that was allowed on a type, and thus had a
separate way to propagate the CC.
These issues were reported
[here](https://github.com/llvm/llvm-project/issues/161077) and
[here](https://github.com/llvm/llvm-project/pull/161349).
Closes: https://github.com/llvm/llvm-project/issues/161077
---------
Signed-off-by: Sarnie, Nick <nick.sarnie@intel.com>
`UnqualPtrTy` didn't always match `llvm::PointerType::getUnqual`:
sometimes it returned a pointer that is not in address space 0 (notably
for SPIRV).
Since `UnqualPtrTy` was used as the "generic" or "default" pointer type,
this patch renames it to `DefaultPtrTy` to avoid confusion with LLVM's
`PointerType::getUnqual`.
This PR adds basic frontend support for `__cluster_dims__` and
`__no_cluster__` attribute.
In CUDA/HIP programming, the ``__cluster_dims__`` attribute can be
applied to a kernel function to set the dimensions of a thread block
cluster. The ``__no_cluster__`` attribute can be applied to a kernel
function to indicate that the thread block cluster feature will not be
enabled at both compile time and kernel launch time. Note that
`__no_cluster__` is a LLVM/Clang only attribute.
Co-authored-by: Yaxun (Sam) Liu <yaxun.liu@amd.com>
Co-authored-by: Jay Foad <jay.foad@amd.com>
This rename was made as part of
https://github.com/llvm/llvm-project/pull/147835 in order to ease
rebasing the PR, and give a nice window for other patches to get rebased
as well.
It has been a while already, so lets go ahead and rename it back.
LLVM models ConstantPointerNull as all-zero, but some GPUs (e.g. AMDGPU
and our downstream GPU target) use a non-zero sentinel for null in
private / local address spaces.
SPIR-V is a supported input for our GPU target. This PR preserves a
canonical zero form in the generic AS while allowing later lowering to
substitute the target’s real sentinel.
In commit e8a486ea9789, a change was made so that certain 256-bit and
512-bit CXX records would be returned in memory, fixing a violation of
the x86-64 psABI (where they had been incorrectly returned in AVX
registers). For compatibility reasons, we want to suppress that ABI-fix
on PlayStation. This commit suppresses that change for PlayStation, and
updates the test to include checking the 512-bit case.
This is part 1 of implementing the typed buffer counters proposal:
https://github.com/llvm/wg-hlsl/blob/main/proposals/0023-typed-buffer-counters.md
This patch adds the initial plumbing for supporting counter variables
associated with structured buffers for the SPIR-V backend. It introduces
an `IsCounter` attribute to `HLSLAttributedResourceType` and threads it
through the AST, type printing, and mangling. It also adds a
`__counter_handle` member to the relevant buffer types in
`HLSLBuiltinTypeDeclBuilder`.
Contributes to https://github.com/llvm/llvm-project/issues/137032
Pad argument registers to preserve overaligned structs in LLVM IR.
Additionally, since i128 values will be lowered as split i64 pairs in
the backend, correctly set the alignment of such arguments as 16 bytes.
This should make clang compliant with the ABI specification and fix
https://github.com/llvm/llvm-project/issues/144709.
Summary:
The vector target should be able to handle vector sizes that are
multiples of the native size, this is useful for implementing math
routines that want to temporarily use a high precision for example.
However, currently this will emit a warning on x86 if any function calls
are involved. https://godbolt.org/z/dK7hGndYh.
I believe that we should be able to relax the ABI restriction if the
functions are completely internal and there were no explicitly states
attributes to conflict. Because the ABI is not exported outside the TU
we should be safe to assume that it won't bite us. In the case that one
call has no features and other does, that will still cause an error. I
may be wrong on this assumption however.
Fixes: https://github.com/llvm/llvm-project/issues/128361
In SPIR-V, kernel arguments are not allowed to be in the Generic AS, in
both Intel's internal SPIR-V offloading implementation as well as
HIPSPV, `CrossWorkgroup` AS1 is used. Do the same for OMPSPV.
Currently with Generic AS the `llvm-spirv` translator blows up if we are
using it, and if not, the GPU runtime blows up.
To get the existing logic to set the correct AS to kick in, we need to
know if the function is a kernel or not at the time we first create the
function that may end up as the kernel.
I use the existing `arrangeSYCLKernelCallerDeclaration` function to do
the right kernel ABI computation, but since the function is not specific
to SYCL anymore because I merged all the device kernel clang attributes
into one.
Rename the function to be accurate to the current behavior,
`arrangeDeviceKernelCallerDeclaration`.
---------
Signed-off-by: Sarnie, Nick <nick.sarnie@intel.com>
The 256-bit maximum vector register size control was removed from AVX10
whitepaper, ref: https://cdrdv2.intel.com/v1/dl/getContent/784343
We have warned these options in LLVM21 through #132542. This patch
removes underlying implementations in LLVM22.
This option allows users to control the image format used for HLSL
resources
when targeting SPIR-V. When the option is enabled, the unknown image
format
is used. Otherwise, the image format is guessed based on the input type.
Fixes https://github.com/llvm/llvm-project/issues/148270
Upgrade the !"grid_constant" !nvvm.annotation to a "nvvm.grid_constant"
attribute. This attribute is much simpler for front-ends to apply and
faster and simpler to query.
This reintroduces `Type.h`, having earlier been renamed to `TypeBase.h`,
as a redirection to `TypeBase.h`, and redirects most users to include
the former instead.
This is a preparatory patch for being able to provide inline definitions
for `Type` methods which would otherwise cause a circular dependency
with `Decl{,CXX}.h`.
Doing these operations into their own NFC patch helps the git rename
detection logic work, preserving the history.
This patch makes clang just a little slower to build (~0.17%), just
because it makes more code indirectly include `DeclCXX.h`.
This is a preparatory patch, to be able to provide inline definitions
for `Type` functions which depend on `Decl{,CXX}.h`. As the latter also
depends on `Type.h`, this would not be possible without some
reorganizing.
Splitting this rename into its own patch allows git to track this as a
rename, and preserve all git history, and not force any code
reformatting.
A later NFC patch will reintroduce `Type.h` as redirection to
`TypeBase.h`, rewriting most places back to directly including `Type.h`
instead of `TypeBase.h`, leaving only a handful of places where this is
necessary.
Then yet a later patch will exploit this by making more stuff inline.
This changes a bunch of places which use getAs<TagType>, including
derived types, just to obtain the tag definition.
This is preparation for #155028, offloading all the changes that PR used
to introduce which don't depend on any new helpers.
This patch handle struct of fixed vector and struct of array of fixed
vector correctly for VLS calling convention in EmitFunctionProlog,
EmitFunctionEpilog and EmitCall.
stack on: https://github.com/llvm/llvm-project/pull/147173
This option is confusingly named. What it actually controls is whether,
under the default of `-ffloat16-excess-precision=standard`, it is
beneficial for performance to perform calculations on float (without
intermediate rounding) or not. For `-ffloat16-excess-precision=none` the
LLVM `half` type will always be used, and all backends are expected to
legalize it correctly.
This is a major change on how we represent nested name qualifications in
the AST.
* The nested name specifier itself and how it's stored is changed. The
prefixes for types are handled within the type hierarchy, which makes
canonicalization for them super cheap, no memory allocation required.
Also translating a type into nested name specifier form becomes a no-op.
An identifier is stored as a DependentNameType. The nested name
specifier gains a lightweight handle class, to be used instead of
passing around pointers, which is similar to what is implemented for
TemplateName. There is still one free bit available, and this handle can
be used within a PointerUnion and PointerIntPair, which should keep
bit-packing aficionados happy.
* The ElaboratedType node is removed, all type nodes in which it could
previously apply to can now store the elaborated keyword and name
qualifier, tail allocating when present.
* TagTypes can now point to the exact declaration found when producing
these, as opposed to the previous situation of there only existing one
TagType per entity. This increases the amount of type sugar retained,
and can have several applications, for example in tracking module
ownership, and other tools which care about source file origins, such as
IWYU. These TagTypes are lazily allocated, in order to limit the
increase in AST size.
This patch offers a great performance benefit.
It greatly improves compilation time for
[stdexec](https://github.com/NVIDIA/stdexec). For one datapoint, for
`test_on2.cpp` in that project, which is the slowest compiling test,
this patch improves `-c` compilation time by about 7.2%, with the
`-fsyntax-only` improvement being at ~12%.
This has great results on compile-time-tracker as well:
This patch also further enables other optimziations in the future, and
will reduce the performance impact of template specialization resugaring
when that lands.
It has some other miscelaneous drive-by fixes.
About the review: Yes the patch is huge, sorry about that. Part of the
reason is that I started by the nested name specifier part, before the
ElaboratedType part, but that had a huge performance downside, as
ElaboratedType is a big performance hog. I didn't have the steam to go
back and change the patch after the fact.
There is also a lot of internal API changes, and it made sense to remove
ElaboratedType in one go, versus removing it from one type at a time, as
that would present much more churn to the users. Also, the nested name
specifier having a different API avoids missing changes related to how
prefixes work now, which could make existing code compile but not work.
How to review: The important changes are all in
`clang/include/clang/AST` and `clang/lib/AST`, with also important
changes in `clang/lib/Sema/TreeTransform.h`.
The rest and bulk of the changes are mostly consequences of the changes
in API.
PS: TagType::getDecl is renamed to `getOriginalDecl` in this patch, just
for easier to rebasing. I plan to rename it back after this lands.
Fixes#136624
Fixes https://github.com/llvm/llvm-project/issues/43179
Fixes https://github.com/llvm/llvm-project/issues/68670
Fixes https://github.com/llvm/llvm-project/issues/92757
Currently, clang coerces (u)int128_t to two i64 IR parameters when they
are passed in registers. This leads to broken debug info for them after
applying SROA+InstCombine. SROA generates IR like this
([godbolt](https://godbolt.org/z/YrTa4chfc)):
```llvm
define dso_local { i64, i64 } @add(i64 noundef %a.coerce0, i64 noundef %a.coerce1) {
entry:
%a.sroa.2.0.insert.ext = zext i64 %a.coerce1 to i128
%a.sroa.2.0.insert.shift = shl nuw i128 %a.sroa.2.0.insert.ext, 64
%a.sroa.0.0.insert.ext = zext i64 %a.coerce0 to i128
%a.sroa.0.0.insert.insert = or i128 %a.sroa.2.0.insert.shift, %a.sroa.0.0.insert.ext
#dbg_value(i128 %a.sroa.0.0.insert.insert, !17, !DIExpression(), !18)
// ...
!17 = !DILocalVariable(name: "a", arg: 1, scope: !10, file: !11, line: 1, type: !14)
// ...
```
and InstCombine then removes the `or`, moving it into the
`DIExpression`, and the `shl` at which point the debug info salvaging in
`Transforms/Local` replaces the arguments with `poison` as it does not
allow constants larger than 64 bit in `DIExpression`s.
I'm working under the assumption that there is interest in fixing this.
If not, please tell me.
By not coercing `int128_t`s into `{i64, i64}` but keeping them as
`i128`, the debug info stays intact and SelectionDAG then generates two
`DW_OP_LLVM_fragment` expressions for the two corresponding argument
registers.
Given that the ABI code for x64 seems to not coerce the argument when it
is passed on the stack, it should not lead to any problems keeping it as
an `i128` when it is passed in registers.
Alternatively, this could be fixed by checking if a constant value fits
in 64 bits in the debug info salvaging code and then extending the value
on the expression stack to the necessary width. This fixes InstCombine
breaking the debug info but then SelectionDAG removes the expression and
that seems significantly more complex to debug.
Another fix may be to generate `DW_OP_LLVM_fragment` expressions when
removing the `or` as it gets marked as disjoint by InstCombine. However,
I don't know if the KnownBits information is still available at the time
the `or` gets removed and it would probably require refactoring of the
debug info salvaging code as that currently only seems to replace single
expressions and is not designed to support generating new debug records.
Converting `(u)int128_t` arguments to `i128` in the IR seems like the
simpler solution, if it doesn't cause any ABI issues.
