Add new elementwise popcount builtin to support HLSL function
'countbits'.
elementwise popcount only accepts integer types.
Add hlsl intrinsic 'countbits'
Closes#99094
Update mma tests in prep for changes needed in a followup patch for
https://github.com/llvm/llvm-project/issues/107229.
Checks for ``clang/test/CodeGen/PowerPC/builtins-ppc-pair-mma-types.c``
seem to have been manually upated to rename temp variables even though
it says checks was auto generated. Regenerate via script.
Add noopt checks for
``clang/test/CodeGen/PowerPC/builtins-ppc-build-pair-mma.c``.
This implements our original design now that LLVM is comfortable with
structs and arrays of scalable vector types. All SVE ACLE intrinsics
already use struct types so the effect of this change is purely the
types used for alloca and function parameters.
There should be no C/C++ user visible change with this patch.
This patch enable the function multiversion(FMV) and `target_clones`
attribute for RISC-V target.
The proposal of `target_clones` syntax can be found at the
https://github.com/riscv-non-isa/riscv-c-api-doc/pull/48 (which has
landed), as modified by the proposed
https://github.com/riscv-non-isa/riscv-c-api-doc/pull/85 (which adds the
priority syntax).
It supports the `target_clones` function attribute and function
multiversioning feature for RISC-V target. It will generate the ifunc
resolver function for the function that declared with target_clones
attribute.
The resolver function will check the version support by runtime object
`__riscv_feature_bits`.
For example:
```
__attribute__((target_clones("default", "arch=+ver1", "arch=+ver2"))) int bar() {
return 1;
}
```
the corresponding resolver will be like:
```
bar.resolver() {
__init_riscv_feature_bits();
// Check arch=+ver1
if ((__riscv_feature_bits.features[0] & BITMASK_OF_VERSION1) == BITMASK_OF_VERSION1) {
return bar.arch=+ver1;
} else {
// Check arch=+ver2
if ((__riscv_feature_bits.features[0] & BITMASK_OF_VERSION2) == BITMASK_OF_VERSION2) {
return bar.arch=+ver2;
} else {
// Default
return bar.default;
}
}
}
```
These transforms all perform a variant of (gep (gep p, x), y)
to (gep p, (x + y)). We can preserve both inbounds and nuw
during such transforms (https://alive2.llvm.org/ce/z/Stu4cN), but
not nusw, which would require proving that the new add is nsw.
For the constant offset case, I've conservatively retained the
logic that checks for negative intermediate offsets, though I'm
not sure it's still reachable nowadays.
Update codegen for func param with transparent_union attr to be that of
the first union member.
This is a followup to #101738 to fix non-ppc codegen and closes#76773.
This patch implements the intrinsics of the form
floatNxM_t vamin[q]_fN(floatNxM_t vn, floatNxM_t vm);
floatNxM_t vamax[q]_fN(floatNxM_t vn, floatNxM_t vm);
as defined in https://github.com/ARM-software/acle/pull/324
---------
Co-authored-by: Hassnaa Hamdi <hassnaa.hamdi@arm.com>
This patch moves NEON immediate argument specification and checking to
the system currently shared by both SVE and SME.
In its current form, the TableGen definition of a NEON intrinsic cannot
control how its immediate arguments are range-checked, this information
must be inferred from the name of the intrinsic by NeonEmitter, which
also assumes that any NEON instruction will only ever receive a single
immediate argument. For SVE/SME instrinsics, this information is more
conveniently supplied in the TableGen definition.
As a result, for each immediate argument, NEON instructions must define
- The index of the immediate argument to be checked
- The type of immediate range check to be performed,
(e.g., ImmCheckShiftRight)
- The index of the argument whose type defines the context
of this immediate check (base type, vector size).
- **Difference from SVE/SME** If this definition generates a polymorphic
NEON builtin, the base type defined by this argument is overwritten by
that of the type code supplied to the overloaded builtin call. This
third argument is omitted in some cases due to this.
Here is an example for
[`vfma_laneq`](https://developer.arm.com/architectures/instruction-sets/intrinsics/#f:@navigationhierarchiessimdisa=[Neon]&q=vfma_laneq)
- The immediate is supplied in argument 3
- The immediate is used as an index into the lanes of argument 2
- So we must perform an immediate check on argument 3, based on the type
information of argument 2.
- `ImmCheck<3, ImmCheckLaneIndex, 2>`
During this work, we discovered that the existing immediate
range-checking system was largely untested, which made it difficult to
make reliable progress. Missing tests have been added to verify this
implementation against all intrinsics which take constrained immediate
arguments. All test immediate range checking tests for NEON intrinsics
are moved to a dedicated directory
`clang/test/Sema/aarch64-neon-immediate-ranges/`.
Add nuw attribute to inbounds GEPs where the expression used to form the
GEP is an addition of unsigned indices.
Relands #105496, which was reverted because it exposed a miscompilation
arising from #98608. This is now fixed by #106512.
Remove flag that turns on the PGOForceFunctionAttrs pass and always add
it to default pipelines when using PGO.
This is NFC by default since PGOOpt->ColdOptType is by default
ColdFuncOpt::Default.
Remove -O2 RUN line in basic.ll since we now have the pipeline tests.
Summary:
This patch proposes new llvm types for RISCV vector tuples represented
as `TargetExtType` which contains both `LMUL` and `NF`(num_fields)
information and keep it all the way down to `selectionDAG` to match the
corresponding `MVT`(support in the following patch).
Detail:
Currently we have built-in C types for RISCV vector tuple type, e.g.
`vint32m1x2_t`, however it's is represented as structure of scalable
vector types, i.e. `{<vscale x 2 x i32>, <vscale x 2 x i32>}`. It loses
the information for num_fields(NF) as struct is flattened during
`selectionDAG`, thus it makes it not possible to handle inline assembly
of vector tuple type, it also makes the calling convention of vector
tuple types handing not strait forward and hard to realize the
allocation code, i.e. `RVVArgDispatcher`.
The llvm IR for the example above is then represented as
`target("riscv.vector.tuple", <vscale x 8 x i8>, 2)` in which the first
type parameter is the equivalent size scalable vecotr of i8 element
type, the following integer parameter is the `NF` of the tuple.
The new RISCV specific vector insert/extract intrinsics are also added
as `llvm.riscv.vector.insert` and `llvm.riscv.vector.extract` to handle
tuple type subvector insertion/extraction since the generic ones only
operates on `VectorType` but not `TargetExtType`.
There are total of 32 llvm types added for each `VREGS * NF <= 8`, where
`VREGS` is the vector registers needed for each `LMUL` and `NF` is
num_fields.
The name of types are:
```
target("riscv.vector.tuple", <vscale x 1 x i8>, 2) // LMUL = mf8, NF = 2
target("riscv.vector.tuple", <vscale x 1 x i8>, 3) // LMUL = mf8, NF = 3
target("riscv.vector.tuple", <vscale x 1 x i8>, 4) // LMUL = mf8, NF = 4
target("riscv.vector.tuple", <vscale x 1 x i8>, 5) // LMUL = mf8, NF = 5
target("riscv.vector.tuple", <vscale x 1 x i8>, 6) // LMUL = mf8, NF = 6
target("riscv.vector.tuple", <vscale x 1 x i8>, 7) // LMUL = mf8, NF = 7
target("riscv.vector.tuple", <vscale x 1 x i8>, 8) // LMUL = mf8, NF = 8
target("riscv.vector.tuple", <vscale x 2 x i8>, 2) // LMUL = mf4, NF = 2
target("riscv.vector.tuple", <vscale x 2 x i8>, 3) // LMUL = mf4, NF = 3
target("riscv.vector.tuple", <vscale x 2 x i8>, 4) // LMUL = mf4, NF = 4
target("riscv.vector.tuple", <vscale x 2 x i8>, 5) // LMUL = mf4, NF = 5
target("riscv.vector.tuple", <vscale x 2 x i8>, 6) // LMUL = mf4, NF = 6
target("riscv.vector.tuple", <vscale x 2 x i8>, 7) // LMUL = mf4, NF = 7
target("riscv.vector.tuple", <vscale x 2 x i8>, 8) // LMUL = mf4, NF = 8
target("riscv.vector.tuple", <vscale x 4 x i8>, 2) // LMUL = mf2, NF = 2
target("riscv.vector.tuple", <vscale x 4 x i8>, 3) // LMUL = mf2, NF = 3
target("riscv.vector.tuple", <vscale x 4 x i8>, 4) // LMUL = mf2, NF = 4
target("riscv.vector.tuple", <vscale x 4 x i8>, 5) // LMUL = mf2, NF = 5
target("riscv.vector.tuple", <vscale x 4 x i8>, 6) // LMUL = mf2, NF = 6
target("riscv.vector.tuple", <vscale x 4 x i8>, 7) // LMUL = mf2, NF = 7
target("riscv.vector.tuple", <vscale x 4 x i8>, 8) // LMUL = mf2, NF = 8
target("riscv.vector.tuple", <vscale x 8 x i8>, 2) // LMUL = m1, NF = 2
target("riscv.vector.tuple", <vscale x 8 x i8>, 3) // LMUL = m1, NF = 3
target("riscv.vector.tuple", <vscale x 8 x i8>, 4) // LMUL = m1, NF = 4
target("riscv.vector.tuple", <vscale x 8 x i8>, 5) // LMUL = m1, NF = 5
target("riscv.vector.tuple", <vscale x 8 x i8>, 6) // LMUL = m1, NF = 6
target("riscv.vector.tuple", <vscale x 8 x i8>, 7) // LMUL = m1, NF = 7
target("riscv.vector.tuple", <vscale x 8 x i8>, 8) // LMUL = m1, NF = 8
target("riscv.vector.tuple", <vscale x 16 x i8>, 2) // LMUL = m2, NF = 2
target("riscv.vector.tuple", <vscale x 16 x i8>, 3) // LMUL = m2, NF = 3
target("riscv.vector.tuple", <vscale x 16 x i8>, 4) // LMUL = m2, NF = 4
target("riscv.vector.tuple", <vscale x 32 x i8>, 2) // LMUL = m4, NF = 2
```
RFC:
https://discourse.llvm.org/t/rfc-support-riscv-vector-tuple-type-in-llvm/80005
The PR https://github.com/llvm/llvm-project/pull/105996 broke taking the
address of a vector:
**compound-literal.c**
```C
typedef int v4i32 __attribute((vector_size(16)));
v4i32 *y = &(v4i32){1,2,3,4};
```
That because the current interpreter handle vector unary operator as a
fallback when the generic code path fail. but the new interpreter was
not. we need to handle `UO_AddrOf` in
`Compiler<Emitter>::VisitVectorUnaryOperator`.
Signed-off-by: yronglin <yronglin777@gmail.com>
This is a partial revert of c66e1d6f3429. Even though that
allowed us to declare v9.2-a support without picking up SVE2
in both the backend and the driver, the frontend itself still
enabled SVE via the arch version's default extensions.
Avoid that by reverting back to v8.7-a while we look into
longer-term solutions.
Commit 0d527e56a5ee ("GlobalIFunc: Make ifunc respect function address
spaces") added support for this within LLVM, but Clang does not properly
honour the target's address spaces when creating IFUNCs, crashing with
RAUW and verifier assertion failures when compiling C code on a target
with a non-zero program address space, so fix this.
Certain intrinsics map to builtins that require an immediate (literal)
argument; make sure we report non-literal arguments.
This has been kicking around downstream for a while, and the recent
removal of the MMX builtins caused me to notice it again.
From @vitalybuka's review on
https://github.com/llvm/llvm-project/pull/104889:
- [x] remove unused variable in tests
- [x] rename `post-decr-while` --> `unsigned-post-decr-while`
- [x] split `add-overflow-test` into `add-unsigned-overflow-test` and
`add-signed-overflow-test`
- [x] be more clear about defaults within docs
- [x] add table to docs
Here's a screenshot of the rendered table so you don't have to build the
html docs yourself to inspect the layout:
CCs: @vitalybuka
---------
Signed-off-by: Justin Stitt <justinstitt@google.com>
Co-authored-by: Vitaly Buka <vitalybuka@google.com>
`x86_64-sie-ps5` is the triple we share with PS5 toolchain users who
have reason to care about such things. The vast majority of PS5 checks
and tests already use this variant. Quashing the handful of stragglers
will help prevent future copy+paste of the discouraged variant.
Introduce the `-fsanitize=realtime` flag in clang driver
Plug in the RealtimeSanitizer PassManager pass in Codegen, and attribute
a function based on if it has the `[[clang::nonblocking]]` function
effect.
This commit introduces attribute bpf_fastcall to declare BPF functions
that do not clobber some of the caller saved registers (R0-R5).
The idea is to generate the code complying with generic BPF ABI,
but allow compatible Linux Kernel to remove unnecessary spills and
fills of non-scratched registers (given some compiler assistance).
For such functions do register allocation as-if caller saved registers
are not clobbered, but later wrap the calls with spill and fill
patterns that are simple to recognize in kernel.
For example for the following C code:
#define __bpf_fastcall __attribute__((bpf_fastcall))
void bar(void) __bpf_fastcall;
void buz(long i, long j, long k);
void foo(long i, long j, long k) {
bar();
buz(i, j, k);
}
First allocate registers as if:
foo:
call bar # note: no spills for i,j,k (r1,r2,r3)
call buz
exit
And later insert spills fills on the peephole phase:
foo:
*(u64 *)(r10 - 8) = r1; # Such call pattern is
*(u64 *)(r10 - 16) = r2; # correct when used with
*(u64 *)(r10 - 24) = r3; # old kernels.
call bar
r3 = *(u64 *)(r10 - 24); # But also allows new
r2 = *(u64 *)(r10 - 16); # kernels to recognize the
r1 = *(u64 *)(r10 - 8); # pattern and remove spills/fills.
call buz
exit
The offsets for generated spills/fills are picked as minimal stack
offsets for the function. Allocated stack slots are not used for any
other purposes, in order to simplify in-kernel analysis.
This fixes a problem which caused clang to assert in the Sema pragma
handling if it encountered "#pragma STDC FP_CONTRACT DEFAULT" when
compiling with the -ffp-contract=fast-honor-pragmas option.
This fixes https://github.com/llvm/llvm-project/issues/104830
The idea behind this canonicalization is that it allows us to handle less
patterns, because we know that some will be canonicalized away. This is
indeed very useful to e.g. know that constants are always on the right.
However, this is only useful if the canonicalization is actually
reliable. This is the case for constants, but not for arguments: Moving
these to the right makes it look like the "more complex" expression is
guaranteed to be on the left, but this is not actually the case in
practice. It fails as soon as you replace the argument with another
instruction.
The end result is that it looks like things correctly work in tests,
while they actually don't. We use the "thwart complexity-based
canonicalization" trick to handle this in tests, but it's often a
challenge for new contributors to get this right, and based on the
regressions this PR originally exposed, we clearly don't get this right
in many cases.
For this reason, I think that it's better to remove this complexity
canonicalization. It will make it much easier to write tests for
commuted cases and make sure that they are handled.
This patch creates an additional EmitRISCVCpuSupports function to handle
situations with multiple features. It also modifies the original
EmitRISCVCpuSupports function to invoke the new one.
With -fsanitize-cfi-icall-experimental-normalize-integers, Clang
appends ".normalized" to KCFI types in CodeGenModule::CreateKCFITypeId,
which changes type hashes also for functions that don't have integer
types in their signatures. However, llvm::setKCFIType does not take
integer normalization into account, which means LLVM generated
functions with KCFI types, e.g. sanitizer constructors, will fail KCFI
checks when integer normalization is enabled in Clang.
Add a cfi-normalize-integers module flag to indicate integer
normalization is used, and append ".normalized" to KCFI types also in
llvm::setKCFIType to fix the type mismatch.
Introduce "-fsanitize-undefined-ignore-overflow-pattern=" which can
be used to disable sanitizer instrumentation for common overflow-dependent
code patterns.
For a wide selection of projects, proper overflow sanitization could
help catch bugs and solve security vulnerabilities. Unfortunately, in
some cases the integer overflow sanitizers are too noisy for their users
and are often left disabled. Providing users with a method to disable
sanitizer instrumentation of common patterns could mean more projects
actually utilize the sanitizers in the first place.
One such project that has opted to not use integer overflow (or
truncation) sanitizers is the Linux Kernel. There has been some
discussion[1] recently concerning mitigation strategies for unexpected
arithmetic overflow. This discussion is still ongoing and a succinct
article[2] accurately sums up the discussion. In summary, many Kernel
developers do not want to introduce more arithmetic wrappers when
most developers understand the code patterns as they are.
Patterns like:
if (base + offset < base) { ... }
or
while (i--) { ... }
or
#define SOME -1UL
are extremely common in a code base like the Linux Kernel. It is
perhaps too much to ask of kernel developers to use arithmetic wrappers
in these cases. For example:
while (wrapping_post_dec(i)) { ... }
which wraps some builtin would not fly. This would incur too many
changes to existing code; the code churn would be too much, at least too
much to justify turning on overflow sanitizers.
Currently, this commit tackles three pervasive idioms:
1. "if (a + b < a)" or some logically-equivalent re-ordering like "if (a > b + a)"
2. "while (i--)" (for unsigned) a post-decrement always overflows here
3. "-1UL, -2UL, etc" negation of unsigned constants will always overflow
The patterns that are excluded can be chosen from the following list:
- add-overflow-test
- post-decr-while
- negated-unsigned-const
These can be enabled with a comma-separated list:
-fsanitize-undefined-ignore-overflow-pattern=add-overflow-test,negated-unsigned-const
"all" or "none" may also be used to specify that all patterns should be
excluded or that none should be.
[1] https://lore.kernel.org/all/202404291502.612E0A10@keescook/
[2] https://lwn.net/Articles/979747/
CCs: @efriedma-quic @kees @jyknight @fmayer @vitalybuka
Signed-off-by: Justin Stitt <justinstitt@google.com>
Co-authored-by: Bill Wendling <morbo@google.com>
