Model zext i1 %x to in as select i1 %x, in 1, in 0 in case, if there are
other select instructions, which can be combined into a bundle.
Fixes#178403
Reviewers: hiraditya, RKSimon
Pull Request: https://github.com/llvm/llvm-project/pull/180635
When merging `.bss` into a code section (e.g., `/MERGE:.bss=.text`), the
INT3 gap-filling loop in `writeSections()` would write past the output
buffer. This happens because `.bss` chunks have `hasData=false`, so they
contribute to `VirtualSize` but not `SizeOfRawData`. The loop was using
chunk RVAs without checking if they exceeded the raw data region.
This caused a crash on Windows with `/FILEALIGN:1` (access violation
0xC0000005). The tight alignment leaves no slack in the mapped buffer,
so the overflow immediately hits unmapped memory.
The fix bounds all memset operations to `rawSize` and exits early when
encountering chunks beyond the raw data boundary.
Fixes#180406
When a global variable has a size that exceeds the size of the address
space it resides in, the verifier should fail as the variable can
neither be materialized nor fully accessed. This patch adds a check to
the verifier to enforce it.
---------
Signed-off-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
Co-authored-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
We're currently unwrapping `less<T>` even if the `key_type` isn't `T`.
This causes the removal of an implicit conversion to `const T&` if the
types mismatch. Making `less<T>` transparent in that case changes
overload resolution and makes it fail potentially.
Fixes#179319
No test because I'm not sure how to reproduce this, but this patch fixes
`CodeGen/ptrauth-qualifier-function.c`.
For function pointer types and function reference types, we use
`Pointer`s these days, so we _can_ return them.
These NodeBuilder classes were deleted from the codebase in 2011
(fifteen years ago!) by 3eae33412d18c4a4a4a8592898b3e65ad5946a89 so
don't declare them as friends.
Fixes following error on macOS:
```
CHECK-NEXT: (llvm::SmallVector<int, 2>) value = size=2 {
^
<stdin>:41:65: note: scanning from here
(llvm::Expected<llvm::SmallVector<int, 2> &>) ExpectedVecRef = {
^
<stdin>:42:31: note: possible intended match here
(std::__1::reference_wrapper<llvm::SmallVector<int, 2> >::type) value = size=2 {
```
To reduce the noise in #134330 for `libcxx/include/__tree`, we migrate
from certain `static_cast` to `std::__static_fancy_pointer_cast` in this
separate patch.
This change is needed to properly work with fancy pointers like
`min_pointer` during constant evaluation for `std::map`
Co-authored-by: Joseph Huber <huberjn@outlook.com>
Fixes#177501
This PR fixes a compilation crash when using character types in OpenMP
REDUCTION clauses with declare_reduction directives.
The problem was that character types weren't being handled properly
during OpenMP lowering. Specifically:
- Missing character length parameters in hlfir.declare operations
- Incorrect type wrapping for by-ref reductions
- Missing special case handling for boxed/unboxed character types
The fix ensures character types are treated similarly to derived types
throughout the reduction pipeline, since fir::isa_trivial() excludes
them.
Added a regression test to verify the fix works for both allocatable and
non-allocatable character reductions.
<img width="654" height="47" alt="image"
src="https://github.com/user-attachments/assets/cc962f01-3432-44ce-befb-324644767c8b"
/>
[CIR] Refactor tests for SVE svdup builtins
Refactor the SVE svdup builtin tests to focus on aspects that are unique to
their code generation: namely, that the expected LLVM SVE intrinsic (or
intrinsics) is emitted. Other codegen details (such as stack allocations
or temporary materialization) are intentionally not checked, as they are
not part of the builtin-specific codegen logic, but rather generic
Clang/CIR lowering behavior.
The generated CIR remains unchanged, but the CHECK lines are simplified
to only match the intrinsic calls, e.g.:
```mlir
cir.call_llvm_intrinsic "aarch64.sve.<intrinsic-name>"
```
For the LLVM IR checks, the tests now run `opt -passes=sroa` to eliminate
irrelevant IR noise. This allows the checks to be reduced to the essential
intrinsic calls, for example:
```llvm
define dso_local <vscale x 2 x double> @test_svdup_n_f64_z(
<vscale x 16 x i1> %0, double %1) {
%3 = call <vscale x 2 x i1>
@llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> %0)
%4 = call <vscale x 2 x double>
@llvm.aarch64.sve.dup.nxv2f64(
<vscale x 2 x double> zeroinitializer, <vscale x 2 x i1> %3, double %1)
ret <vscale x 2 x double> %4
}
```
Sub-reductions can be implemented in two ways:
(1) negate the operand in the vector loop (the default way).
(2) subtract the reduced value from the init value in the middle block.
Note that both ways keep the reduction itself as an 'add' reduction,
which is necessary because only llvm.vector.partial.reduce.add exists.
The ISD nodes for partial reductions don't support folding the
sub/negation into its operands because the following is not a valid
transformation:
```
sub(0, mul(ext(a), ext(b)))
-> mul(ext(a), ext(sub(0, b)))
```
It can therefore be better to choose option (2) such that the partial
reduction is always positive (starting at '0') and to do a final
subtract in the middle block.
For AArch64 there are no dot-product instructions that can
do a `partial.reduce.sub(acc, mul(ext(a), ext(b)))` operation.
I'm not sure if such instructions exist for other targets.
(If so then we may want to make this decision a target option)
This PR also increases the AArch64 cost of a partial sub-reduction
when this exists in an 'add-sub' reduction chain.
Fixes https://github.com/llvm/llvm-project/issues/178703
The first param of LoopRotatePass is EnableHeaderDuplication. The value
'true' means 'enable the header duplication'.
`LoopRotatePass(bool EnableHeaderDuplication, bool PrepareForLTO)`
---------
Co-authored-by: Pengcheng Wang <wangpengcheng.pp@bytedance.com>
This pass splits up the `vscaleRange` pass-option from the
`VScaleAttrPass` into `vscaleMin` and `vscaleMax` respectively, since a
`std::pair<>` cannot be used as a cli-option and crashes when running
`flang -march=rv64gcv -O3 file.f90 -mmlir -debug`.
Since the options can now be set individually I added some error
checking following the semantics described in the langref
https://llvm.org/docs/LangRef.html#function-attributes.
I also added tests since there were none for only this pass before.
The heuristic for deciding which scope line to use for a continuation
funclet relies on iterating on the instructions of the first BB of the
continuation. Often, this contains a single unconditional branch, which
is skipped by the heuristic. However, in coro-retcon, two such
"jump-only" BBs are generated. This patch amends the heuristic to
account for that.
Improve `ScheduleOrderedAssignments` to avoid creating temporary storage
for masks in `WHERE` constructs when the mask modification is "aligned"
with the assignment (e.g., `where(a(i)>0) a(i)=...`).
- Identify "aligned" conflicts (identical array elements accessed in
order) using the `ArraySectionAnalyzer` that is extracted from
OptimizedBufferization.
- Defer saving regions with aligned conflicts, allowing fusion if
possible.
- Implement retroactive saving: if a region was modified in a previous
run (fused via aligned conflict) but is needed by a later split run,
insert a `SaveEntity` action before the modifying run.
- Use `std::list` for the schedule to support stable iterators for run
insertion.
- Update tests to verify fewer temporaries and correct retroactive
saves.
- Update flang pipeline at O2 and more to try fusing assignments in
WHERE/FORALL. This allows maximizing the chances that mask temps are not
needed (because a mask variable cannot be reused in a later run/loop
nest if it was modified even if all the accesses are in order, being in
order only matter for accesses generated inside the same loop nest).
This fixes suboptimal code generation where temporaries were created
unnecessarily for common patterns like `where (x > 0) x = ...`.
Adds rules for G_ATOMICRMW_{MAX, MIN, UMAX, UMIN, UINC_WRAP, UDEC_WRAP}.
Each of these generic opcode are supported for S32 and S64 types
on flat, global and local address spaces.
Add ptrtoaddr tests with address spaces with unstable and external but
stable pointer representations.
Currently we incorrectly form ptrtoaddr for unstsable pointers. See
discussion in https://github.com/llvm/llvm-project/pull/178861 for more
details.
This patch adds support in Clang for the PRFM IR instruction, by adding
the following builtin:
void __pldir(void const *addr);
This builtin is described in the following ACLE proposal:
https://github.com/ARM-software/acle/pull/406
This patch extends the support added in #158088 to loops where the
assignment is non-speculatable (e.g. a conditional load or divide).
For example, the following loop can now be vectorized:
```
int simple_csa_int_load(
int* a, int* b, int default_val, int N, int threshold)
{
int result = default_val;
for (int i = 0; i < N; ++i)
if (a[i] > threshold)
result = b[i];
return result;
}
```
It does this by extending the recurrence matching from only looking for
selects, to include phis where all operands are the header phi, except
for one which can be an arbitrary value outside the recurrence.
---
Reverts llvm/llvm-project#180275 (original PR: #178862)
Additional type legalization for `ISD::VECTOR_FIND_LAST_ACTIVE` was
added in #180290, which should resolve the backend crashes on x86.
This change adds a `RUN` line in vector-transfer-flatten.mlir that will
use `vector.flatten_vector_transfer_ops` that was introduced in #178134.
It also removes a test added in the original PR whose coverage is
already provided by pre-existing tests.
Currently, UniformQuantizedType only supports built-in MLIR storage
types such as Integer. LLM quantization research introducing feature of
using NF4 as a low precision datatype (see
https://arxiv.org/pdf/2305.14314). There is a growing need to make the
system extensible and maintainable as more types are added. Ensuring
that MLIR can natively support NF4 through a clean, extensible interface
is essential for both current and future quantization workflows.
**Current Approach and Its Limitations:**
- The present implementation relies on dynamic checks (e.g., type
switches or if-else chains) to determine the storage type and retrieve
type-specific information for legality checks.
- This approach works for a small, fixed set of types, but as the number
of supported types grows, the code becomes harder to read, maintain, and
extend.
**Proposed Interface-Based Approach:**
- Define a StorageTypeInterface that specifies the required methods any
storage type must implement to be used in UniformQuantizedType.
- Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types
like NF4) would implement this interface, encapsulating their
type-specific logic.
- When UniformQuantizedType needs to check legality or retrieve
information, it can use MLIR’s dyn_cast mechanism to check if the type
implements the interface and then call the required methods.
- This design decouples UniformQuantizedType from the specifics of each
storage type, making it easy to add new types (such as NF4) without
modifying the core logic or introducing more type checks.
**Benefits:**
- Extensibility: New storage types can be added by simply implementing
the interface, without touching the core UniformQuantizedType logic.
- Readability: The code is cleaner, as it avoids large switch statements
or if-else chains.
- Maintainability: Type-specific logic is encapsulated within each type,
reducing the risk of errors and making the codebase easier to understand
and update.
Start trying to use SimplifyDemandedFPClass on instructions, starting
with fmul. This subsumes the old transform on multiply of 0. The
main change is the introduction of nnan/ninf. I do not think anywhere
was systematically trying to introduce fast math flags before, though
a few odd transforms would set them.
Previously we only called SimplifyDemandedFPClass on function returns
with nofpclass annotations. Start following the pattern of
SimplifyDemandedBits, where this will be called from relevant root
instructions.
I was wondering if this should go into InstCombineAggressive, but that
apparently does not make use of InstCombineInternal's worklist.
Fix https://github.com/llvm/llvm-project/issues/180374
I initially suspected an issue with some lower bound adjustment missing,
and indeed found an unrelated issue because gen1DSection was always
called with all-ones lower bounds because the genLowerbounds was called
on the result fir.shape.
But this is actually not relevant for the issue where this code path is
not exercised. The issue was `nuw` (no unsigned-wrap) was being set on
arithmetic inside the kernel generated for CSHIFT, but because this
arithmetic is dealing with user defined bounds, it may actually have to
deal with negative values (even if the offsets from the CSHIFT itself
are not negative).
This caused LLVM optimization to generate completely invalid code when
the lower bounds of CSHIFT input are zero or less.
Now that getModRefInfo for calls handles read and write effects by
examining both calls, the clobbering query no longer needs to treat
reads as clobbers. Update the check to consider writes only, aligning
call handling with other instructions
The `CHECK` for `(int)` was too strict. On macOS the type prints as:
```
08:46:24 29: (lldb) v -T ExpectedRef
08:46:24 30: (llvm::Expected<int &>) ExpectedRef = {
08:46:24 next:14'0 X error: no match found
08:46:24 31: (std::__1::reference_wrapper<int>::type) value = 100
08:46:24 next:14'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
08:46:24 next:14'1 ? possible intended match
08:46:24 32: }
08:46:24 next:14'0 ~~
```
Without alias analysis Flang assumes no aliasing in lowering record
assignments which can result in miscompilation of programs using
`SEQUENCE` types and `EQUIVALENCE`.
Use alias analysis to guard the fast path in `genRecordAssignment`;
otherwise fall back to element-wise expansion.
Update FIR FileCheck expectations
Add `FIRAnalysis` to "flang/unittests/Optimizer/CMakeLists.txt" to fix
the Windows x64 build failure (linker error).
Add `SEQUENCE` handling and update tests accordingly.
Fixes#175246 (and includes the fix to
"flang/lib/Optimizer/Builder/CMakeLists.txt" in PR #176483).
Co-authored-by: Matt P. Dziubinski <matt-p.dziubinski@hpe.com>
Same as https://github.com/llvm/llvm-project/pull/177988, but for
fminimum_num/fmaximum_num. Directly canonicalize these to the
corresponding intrinsics, and let the shrinking happen directly on the
intrinsics.
Allow a to define a set of Types that are not shown by default when
doing default debug loggin (e.g., LIBOMPTARGET_DEBUG=All).
Users can enable output of those types of messages by explicitly adding
them to LIBOMPTARGET_DEBUG.
Used to implement: #180545
---------
Co-authored-by: Michael Klemm <michael.klemm@amd.com>
This lowers the splitting as:
```
any_active(hi_mask)
? (find_last_active(hi_mask) + lo_mask.getVectorElementCount())
: find_last_active(lo_mask)
```
And trivially lowers `<1 x i1>` scalarization to returning zero. Which
is a natural result of the splitting (and the lack of a sentinel
"none-active" result value).
The lowerings likely can be improved. This patch is for completeness.
Should fix:
https://github.com/llvm/llvm-project/pull/178862#issuecomment-3862310334Fixes#180212
