Previous commit message:
>Previous commit message:
>
>> Original commit message:
>>
>>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>>
>>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>>
>>The following commits should fix the failures that arose when I
previously tried to land this commit:
>>
>>
>>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>>
>>
>>600514a637
should address the MLIR failures
>
>The previous commit was reverted with
d23cb79ba4
because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
>
>Those tests failed because MLIR's default SM was 5.0, which caused
NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
>
>243f011577
should address this by bumping MLIR's default SM to 7.5. Now, using
MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
---
The previous commit was reverted with
e9b578a4d77025e18318efedd0f3f3764338d859
[because](https://github.com/llvm/llvm-project/pull/179304#issuecomment-3856301333)
the clang driver set the default PTX ISA version to v4.2 when no CUDA
installation is found. However, given our patch, we should not set a
default; instead, let the LLVM backend select the appropriate PTX ISA
version for the target SM.
This patch adds support in Clang for the PRFM IR instruction, by adding
the following builtin:
void __pldir(void const *addr);
This builtin is described in the following ACLE proposal:
https://github.com/ARM-software/acle/pull/406
Start trying to use SimplifyDemandedFPClass on instructions, starting
with fmul. This subsumes the old transform on multiply of 0. The
main change is the introduction of nnan/ninf. I do not think anywhere
was systematically trying to introduce fast math flags before, though
a few odd transforms would set them.
Previously we only called SimplifyDemandedFPClass on function returns
with nofpclass annotations. Start following the pattern of
SimplifyDemandedBits, where this will be called from relevant root
instructions.
I was wondering if this should go into InstCombineAggressive, but that
apparently does not make use of InstCombineInternal's worklist.
Rather than creating a dedicated ClangIR test file, the original test file for
this intrinsic is effectively reused:
* clang/test/CodeGen/AArch64/neon-intrinsics.c
“Effectively” meaning that the corresponding test is moved (rather than
literally reused) to a new file within the original AArch64 builtins test
directory:
* clang/test/CodeGen/AArch64/neon/intrinsics.c
This is necessary to avoid lowering unsupported examples from intrinsics.c with
`-fclangir`. The new file will eventually replace the original one once all
builtins from it can be lowered via ClangIR.
To facilitate test re-use, new LIT "feature" is added so that CIR tests can be
run conditionally, e.g. the following will only run when `CLANG_ENABLE_CIR` is
set:
```C
// RUN: %if cir %{%clang_cc1 ... %}
```
This sort of substitutions are documented in [2].
REFERENCES:
[1] https://developer.arm.com/architectures/instruction-sets/intrinsics/#f:@navigationhierarchiessimdisa=[Neon]&q=vceqzd_s64
[2] https://llvm.org/docs/TestingGuide.html#substitutions
This reverts commit 047db150c66e245e9df7db178b893ce6b29820f5.
The original version of the commit caused assertion failures in DSE.
Those were fixed in ec059d81aafedb253a02d6f490ad9b9747611038, so trying
to reland this again.
This patch adds initial support for AMD Zen 6 architecture (znver6):
- Added znver6 CPU target recognition in Clang and LLVM
- Updated compiler-rt CPU model detection for znver6
- Added znver6 to target parser and host CPU detection
- Added znver6 to various optimizer tests
znver6 features: FP16, AVXVNNIINT8, AVXNECONVERT, AVXIFMA (without BMM).
This helps to clean up any dead stores that come up at the end of the
destructor. The motivating example was a refactoring in libc++'s
basic_string implementation in 8dae17be2991cd7f0d7fd9aa5aecd064520a14f6
that added a zeroing store into the destructor, causing a large
performance regression on an internal workload. We also saw a ~0.2%
performance increase on an internal server workload when enabling this.
I also tested this against all of the non-flaky tests in our large C++
codebase and found a minimal number of issues that all happened to be in
user code.
This is my second attempt to merge the #33210 fix. My original patch
#178220 caused test failures on risc-v, as I committed a new test file
that was too target-specific. I reverted the PR (#180183) to avoid
causing disruption, as I did not know how long the fix would take.
I have now fixed the test so that it behaves correctly on risc-v. I have
also changed the run lines so it is tested on aarch64, amd64 and risc-v.
Previous commit message:
> Original commit message:
>
>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>
>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>
>The following commits should fix the failures that arose when I
previously tried to land this commit:
>
>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>
>600514a637
should address the MLIR failures
---
The previous commit was reverted with
d23cb79ba497281de050ef609cb91b91058bf323 because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
Those tests failed because MLIR's default SM was 5.0, which caused NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
243f011577193c99358ccc4142b296d4fa80ea11 should address this by bumping
MLIR's default SM to 7.5. Now, using MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
This just added unnecessary work to the IR, since they are only used for
load and store, which just causes some IR noise. Tests updated by UTC
script to remove the extra lines.
Allows __funcref pointers to be used as the element type for WASM tables
in Clang (static, global, zero-length arrays of a reference type).
Modifies `QualType::isWebAssemblyFuncrefType` to correctly look at the
addrspace of the pointee, rather than the pointer type.
Related: #140933
Currently, the AIX linker and loader do not provide a mechanism to
implement ifuncs similar to GNU_ifunc on ELF Linux.
On AIX, we will lower `__attribute__((ifunc("resolver"))` to the llvm
`ifunc` as other platforms do. The llvm `ifunc` in turn will get lowered
at late stages of the optimization pipeline to an AIX-specific
implementation. No special linkage or relocations are needed when
generating assembly/object output.
On AIX, a function `foo` has two symbols associated with it: a function
descriptor (`foo`) residing in the `.data` section, and an entry point
(`.foo`) residing in the `.text` section. The first field of the
descriptor is the address of the entry point. Typically, the address
field in the descriptor is initialized once: statically, at load time
(?), or at runtime if runtime linking is enabled.
Here we would like to use the address field in the descriptor to
implement the `ifunc` semantics. Specifically, the ifunc function will
become a stub that jumps to the entry point in the address field. A
constructor function is linked into every linkage module. The
constructor walks an array of `{descriptor, resolver}` pairs, calling
the resolver and saving the result in the address field in the
descriptor (thus setting `foo`'s descriptor to point to the resolved
version early during program runtime).
Known limitations:
- Due to bug #161576, which affects object generation path, you will
need either `-ffunction-sections` or `-fno-integrated-as` to generate a
correct/linkable object file.
- aliases to ifuncs are not supported, a testcase has been added and
marked XFAIL. I'm planning to address in a follow-up PR because it's not
important enough, IMHO, for this PR
- dead ifuncs in a CU that contains at least one live ifunc, will result
in all ifuncs being kept by the linker. The fix for this is common with
a similar problem we have with PGO. PR #159435 is trying to provide a
mechanism that will allow the ifunc and PGO implementations to avoid the
dead code retention at the link step.
- the resolver must return a function that is in the same DSO as the
ifunc; the compiler will try to detect if this condition is violated and
report it, but it cannot detect it in general. To be safe, all candidate
functions (returned by a particular resolver) must either be static or
have hidden/protected visibility. This is so that the ifunc stub doesn't
have to save and restore the TOC register r2. In future work, this case
will be supported and the requirement will be lifted.
---------
Co-authored-by: Wael Yehia <wyehia@ca.ibm.com>
ptrauth intrinsic to safely construct relative ptr for swift coroutines.
A ptrauth intrinsic for swift co-routine support that allows creation of
signed pointer
from offset stored at address relative to the pointer.
Following C-like pseudo code (ignoring keys,discriminators) explains its
operation:
let rawptr = PACauth(inputptr);
return PACsign( rawptr + signextend64( *(int32*)(rawptr+addend) ))
What: Authenticate a signed pointer, load a 32bit value at offset
'addend' from pointer,
add this value to pointer, sign this new pointer.
builtin: __builtin_ptrauth_auth_load_relative_and_sign
intrinsic: ptrauth_auth_resign_load_relative
This reverts commit e3156c531da5aa4ec604605ed4e19638879d773c.
We need to resolve a crash on trunk and LLVM 22. Reverting makes it
easier to backport.
Fixes#176637.
This attribute is currently only permitted on functions, but sometimes
it can be useful to opt out a specific local variable. That way the
function can still get a stack protector if other locals require it, but
locals which are opted out won't have the overhead of a stack protector
if not needed.
This is paired with an LLVM change which has already landed, which uses
the new metadata to opt specific alloca instructions out of requiring a
stack protector.
I did not replace riscv.clmulh/clmulr since those require a multiple
instruction pattern match. I wanted to ensure that -O0 will select the
correct instructions without relying on combines.
Reverts llvm/llvm-project#177459
`mlir-nvidia` and `mlir-nvidia-gcc7` Buildbots are failing.
The blamelist is small and likely because of my change. Preemptively
reverting.
Original commit message:
> When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>
>When no SM or PTX version is specified, we now use PTX 3.2 (the minimum
for the default SM 3.0) instead of PTX 6.0.
---
The following commits should fix the failures that arose when I
previously tried to land this commit:
- 9fc5fd0ad689eed94f65b1d6d10f9c5642935e68 should address the
`llvm-nvptx*-nvidia-*` build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
- 600514a63760c6730e4cd970d2fcead9c5a897b3 should address the MLIR
failures
- Make v8f16 a legal type so that arguments can be passed in vector
registers. Handle fp16 vectors so that they have the same ABI as other
fp vectors.
- Set the preferred vector action for fp16 vectors to "split". This will
scalarize all operations, which is not always necessary (like with
memory operations), but it avoids the superfluous operations that result
after first widening and then scalarizing a narrow vector (like v4f16).
Fixes#168992
Implement the export pragma that is used in the z/OS XL C/C++ compiler
to indicate that an external symbol is to be exported from the shared
library. The syntax for the pragma is:
```
'#pragma' 'export' '(' name ')'
```
For C++ if `name` is a function it needs to be declared `extern "C"`.
See the following for the XL documentation:
- https://www.ibm.com/docs/en/zos/3.1.0?topic=descriptions-pragma-export
This code was originally in PR
https://github.com/llvm/llvm-project/pull/111035. I have split it out
into separate PRs so the code for #pragma export is in one PR and the
code for _Export keyword is in another. See that original PR for earlier
comments.
Currently we do not emit lifetimes by default when compiling with
memtag-stack - which means we don't catch use-after-scope (when
compiling without optimization).
This patch fixes that by mirroring ASan, HWASan and MSan, and always
emitting lifetime markers. The patch is based on the changes made in
aeca569.
rdar://163713381
Summary:
The `__scoped_atomic` builtins are supposed to match the standard
GNU-flavored `__atomic` builtins. We added a scoped builtin without a
corresponding standard one before the fork so this should be added in
the release candidate. These were originally added in
https://github.com/llvm/llvm-project/pull/168666
Also, the name `uinc_wrap` does not follow the naming convention. The
GNU atomics use `fetch_xyz` to indicate that the builtin returns the
previous location's value as part of the RMW operation, which these do.
This PR renames it and its uses.
Mostly inspired by https://github.com/llvm/llvm-project/pull/152971
CRC32 implementation using reversed polynomial that does not match an
Intel manual, can be changed to canonical implementation if required (if
there is a canonical implementation we should use, please attach a link)
Closes#168881
Part of #30794
This patch makes the dead_on_return parameter attribute optionally require a number
of bytes to be passed in to specify the number of bytes known to be dead
upon function return/unwind. This is aimed at enabling annotating the
this pointer in C++ destructors with dead_on_return in clang. We need
this to handle cases like the following:
```
struct X {
int n;
~X() {
this[n].n = 0;
}
};
void f() {
X xs[] = {42, -1};
}
```
Where we only certain that sizeof(X) bytes are dead upon return of ~X.
Otherwise DSE would be able to eliminate the store in ~X which would not
be correct.
This patch only does the wiring within IR. Future patches will make
clang emit correct sizing information and update DSE to only delete
stores to objects marked dead_on_return that are provably in bounds of
the number of bytes specified to be dead_on_return.
Reviewers: nikic, alinas, antoniofrighetto
Pull Request: https://github.com/llvm/llvm-project/pull/171712
Support the following BCD format conversion builtins for PowerPC.
- `__builtin_bcdshift` – Shifts a packed decimal value by a specified
number of decimal digits.
- `__builtin_bcdshiftround` – Shifts a packed decimal value by a
specified number of decimal digits, with rounding applied.
- `__builtin_bcdtruncate` –Truncates a packed decimal value to a
specified number of digits.
- `__builtin_bcdunsignedtruncate` – Truncates a packed decimal value and
returns the result as an unsigned packed decimal.
- `__builtin_bcdunsignedshift` – Shifts an unsigned packed decimal value
by a specified number of digits.
> Note: This built-in functions are valid only when all following
conditions are met:
> -qarch is set to utilize POWER9 technology.
> The bcd.h file is included.
## Prototypes
```c
vector unsigned char __builtin_bcdshift(vector unsigned char, int, unsigned char);
vector unsigned char __builtin_bcdshiftround(vector unsigned char, int, unsigned char);
vector unsigned char __builtin_bcdtruncate(vector unsigned char, int, unsigned char);
vector unsigned char __builtin_bcdunsignedtruncate(vector unsigned char, int);
vector unsigned char __builtin_bcdunsignedshift(vector unsigned char, int);
```
---------
This patch adds type-generic rotate builtins that accept any unsigned
integer
type. These builtins provide:
- Support for all unsigned integer types, including _BitInt
- Constexpr evaluation capability
- Automatic normalization of rotation counts modulo the bit-width
- Proper handling of negative rotation counts (converted to equivalent
positive rotations in the opposite direction)
- Implicit conversion support for both arguments for
types with conversion operators.
The builtins follow C23 naming conventions.
Resolves https://github.com/llvm/llvm-project/issues/122819
There is no reason to use these over fpext/fptrunc and bitcast.
Split out from #174484. The test coverage is also shockingly bad,
so adds a new wasm test which shows different contexts the intrinsics
are used.
I've also reverted this to a more conservative version that leaves the
useFP16ConversionIntrinsics configuration in place, and only replaces
the exact intrinsic usage. This should be removed, but it seems to have
turned into a buggy ABI option. Some contexts which probably meant to
check NativeHalfType or NativeHalfArgsAndReturns were relying on this
instead. Additionally, some of the SVE intrinsics appear to be using
__fp16 but really expect _Float16 treatment.
Currently, AMDGPU functions have `target-features` attribute populated
with all default features for the target GPU. This is redundant because
the backend can derive these defaults from the `target-cpu` attribute
via `AMDGPUTargetMachine::getFeatureString()`.
In this PR, for AMDGPU targets only:
- Functions without explicit target attributes no longer emit
`target-features`
- Functions with `__attribute__((target(...)))` or `-target-feature`
emit only features that differ from the target's defaults (delta)
The backend already handles missing `target-features` correctly by
falling back to the TargetMachine's defaults.
A new cc1 flag `-famdgpu-emit-full-target-features` is added to emit
full features when needed.
Example:
Before:
```llvm
attributes #0 = { "target-cpu"="gfx90a" "target-features"="+16-bit-insts,+atomic-buffer-global-pk-add-f16-insts,+atomic-fadd-rtn-insts,+ci-insts,+dl-insts,+dot1-insts,+dot2-insts,..." }
```
After (default):
```llvm
attributes #0 = { "target-cpu"="gfx90a" }
```
After (with explicit `+wavefrontsize32` override):
```llvm
attributes #0 = { "target-cpu"="gfx90a" "target-features"="+wavefrontsize32" }
```
This change adds the following missing half-precision
add/sub/fma intrinsics for the NVPTX target:
- `llvm.nvvm.add.rn{.ftz}.sat.f16`
- `llvm.nvvm.add.rn{.ftz}.sat.v2f16`
- `llvm.nvvm.mul.rn{.ftz}.sat.f16`
- `llvm.nvvm.mul.rn{.ftz}.sat.v2f16`
- `llvm.nvvm.fma.rn.oob.*`
We lower `fneg` followed by one of the above addition
intrinsics to the corresponding `sub` instruction.
This also removes some incorrect `bf16` fma intrinsics with no
valid lowering.
PTX spec reference:
https://docs.nvidia.com/cuda/parallel-thread-execution/#half-precision-floating-point-instructions
Summary:
We support variadic functions in AMDGPU / NVPTX via an LLVM-IR pass.
This patch applies the same handling here to support them on this
target.
I am unsure what the ABI should look like here, I have mostly copied the
one we use for NVPTX where it's basically a struct layout with natural
alignment. This wastes some space, which is why AMDGPU does not pad
them.
Additionally, this required allowing the SPIRV_FUNC calling convention.
I'm assuming this is compatible with the C calling convention in IR, but
I will need someone to confirm that for me.
This commit adds 4 Clang builtins for PowerPC AMO store operations:
__builtin_amo_stwat for 32-bit unsigned operations
__builtin_amo_stdat for 64-bit unsigned operations
__builtin_amo_stwat_s for 32-bit signed operations
__builtin_amo_stdat_s for 64-bit signed operations
and maps GCC's AMO store functions to these Clang builtins for
compatibility.
This commit adds 4 Clang builtins for PowerPC AMO load conditional
increment and decrement operations:
__builtin_amo_lwat_cond for 32-bit unsigned operations
__builtin_amo_ldat_cond for 64-bit unsigned operations
__builtin_amo_lwat_cond_s for 32-bit signed operations
__builtin_amo_ldat_cond_s for 64-bit signed operations
This avoids implicit truncation assertions with
https://github.com/llvm/llvm-project/pull/171456. Apparently the
alignment code path was previously untested.
The first commit switches the test to generated check lines.
