For consistency with `ConstantInt::get()`, add an ImplicitTrunc
parameter to `ConstantInt::getSigned()` as well. It currently defaults
to true and will be flipped to false in the future (by #171456).
This change adds full support for the ptx `barrier.cta.red` instruction,
following the same conventions as are already used for
`barrier.cta.sync` and `barrier.cta.arrive`.
In addition this MR removes the following intrinsics which are no longer
needed:
* llvm.nvvm.barrier0.popc -->
llvm.nvvm.barrier.cta.red.popc.aligned.all(0, c)
* llvm.nvvm.barrier0.and -->
llvm.nvvm.barrier.cta.red.and.aligned.all(0, z)
* llvm.nvvm.barrier0.or -->
llvm.nvvm.barrier.cta.red.or.aligned.all(0, z)
IFUNCs require loader support, so for arbitrary environments, the safe
assumption is to assume that they are not supported. In particular,
aarch64-linux-pauthtest may be used with musl, and was wrongly detected
as supporting IFUNCs.
With IFUNC support now being detected more reliably, this also removes
the check for PAuth support. If both are supported, either would work.
This adds parsing and lowering of the COMBINER clause. It utilizes the
existing lowering code for combiner-expression to lower the COMBINER
clause as well.
The pass now contains a non-fp expansion and should
be used for any similar expansions regardless of the
types involved. Hence a generic name seems apt.
Rename the source files, pass, and adjust the pass
description. Move all tests for the expansions
that have previously been merged into the pass
to a single directory.
Implements initial support for vk::push_constant.
As is, this allows handling simple push constants, but has one
main issue: layout can be incorrect (See #168401). The layout
issue being not only push-constant related, it's ignored for this PR.
The frontend part of the implementation is straightforward:
- adding a new attribute
- when targeting vulkan/spirv, we process it
- global variables with this attribute gets a new AS:
hlsl_push_constant
The IR has nothing specific, only some RO globals in this new AS.
On the SPIR-V side, we not convert this AS into a PushConstant storage
class. But this creates some issues: the variables in this storage class
must have a specific set of decoration to define their layout.
Current infra to create the SPIR-V types lacks the context required to
make this decision: no indication on the AS or context around the type
being created. Refactoring this would be a heavy task as it would
require getting this information in every place using the GR for type
creation.
Instead, we do something similar to CBuffers:
- find all globals with this address space, and change their type to
a target-specific type.
- insert a new intrinsic in place of every reference to this global
variable.
This allow the backend to handle both layout variables loads and type
lowering independently.
Type lowering has nothing specific: when we encounter a target extension
type with spirv.PushConstant, we lower this to the correct SPIR-V type
with the proper offset & block decorations.
As for the intrinsic, it's mostly a no-op, but required since we have
this target-specific type.
Note: this implementation prevents the static declaration of multiple
push constants in a single shader module. The actual specification is
more relaxed: there can be only one **used** push constant block per
entrypoint. To correctly implement this, we'd require to keep some
additional state to determine the list of statically used resources per
entrypoint. This shall be addressed as a follow-up (see #170310)
Both passes expand instructions at the IR level.
They use the same kind of instruction visitation
logic and contain significant code duplication e.g.
for scalarization.
Implements `reserveAllocationSpace` and provides an option to enable
`needsToReserveAllocationSpace` for large-memory environments with
AArch64.
The [AArch64
ABI](https://github.com/ARM-software/abi-aa/blob/main/sysvabi64/sysvabi64.rst#7code-models)
has restrictions on the distance between TEXT and GOT sections as the
instructions to reference them are limited to 2 or 4GB. Allocating
sections in multiple blocks can result in distances greater than that on
systems with lots of memory. In those environments several projects
using SectionMemoryManager with MCJIT have run across assertion failures
for the R_AARCH64_ADR_PREL_PG_HI21 instruction as it attempts to address
across distances greater than 2GB (an int32).
Fixes#71963 by allocating all sections in a single contiguous memory
allocation, limiting the distance required for instruction offsets
similar to how pre-compiled binaries would be loaded into memory.
Co-authored-by: Lang Hames <lhames@gmail.com>
This PR implements the emitting of the post-link CFG information in PGO
analysis map, as explained in the
[RFC](https://discourse.llvm.org/t/rfc-extending-the-pgo-analysis-map-with-propeller-cfg-frequencies/88617).
This is enabled by a flag `pgo-analysis-map-emit-bb-sections-cfg`.
This PR bumps the SHT_LLVM_BB_ADDR_MAP version to 5.
Also includes some refactoring changes related to storing the CFG in the
Basic block sections profile reader.
Also renames CWrapperFunctionResult to CWrapperFunctionBuffer.
These types are used as argument buffers, as well as result buffers. The
new name better reflects their purpose, and is consistent with naming in
the new ORC runtime (llvm-project/orc-rt).
The WrapperFunctionBuffer(CWrapperFunctionBuffer) constructor takes
ownership of the underlying buffer (if one exists). Making the
constructor explicit makes this clearer at the call site.
This implements the major piece of
https://discourse.llvm.org/t/rfc-codegen-new-pass-manager-pipeline-construction-design/84659,
making it explicit when we break the function pipeline up.
We essentially get rid of the AddPass and AddMachinePass helpers and
replace them with explicit functions for the pass types. The user then
needs to explicitly call flushFPMstoMPM before breaking.
This is sort of a hybrid of the current construction and what the RFC
proposed. The alternative would be passing around FunctionPassManagers
and having the pipeline actually explicitly constructed. I think this
compromises ergonomics slightly (needing to pass a FPM in many more
places). It is also nice to assert that the function pass manager is
empty when adding a module pass, which is easier when CodegenPassBuilder
owns the FPM and MFPM.
Sometimes we want to create a SchedPredicate out of different types of
SchedPredicate: composing a `FeatureSchedPredicate` with a
`MCSchedPredicate`, for example. For those cases, this patch creates
several new TableGen constructions so that users can create a
SchedPredicate like
```
AllOfSchedPreds<[FeatureFooPred,
MCSchedPredicate<CheckZeroOperand<1>>]>
```
which evaluates to true only when both `FeatureFooPred` and
`MCSchedPredicate<CheckZeroOperand<1>>` hold.
This PR is the first step towards introducing LFI into LLVM as a new
sub-architecture backend of AArch64. For details, please see the
[RFC](https://discourse.llvm.org/t/rfc-lightweight-fault-isolation-lfi-efficient-native-code-sandboxing-upstream-lfi-target-and-compiler-changes/88380),
which has been approved for AArch64.
This patch creates the `aarch64_lfi` architecture, and marks the
appropriate registers as reserved when it is targeted (`x25`, `x26`,
`x27`, `x28`). It also adds a Clang driver toolchain for targeting LFI,
and updates the compiler-rt CMake to allow builds for the `aarch64_lfi`
target. The patch also includes documentation for LFI and the rewrites
that will be implemented in future patches.
I am planning to split the relevant modifications for LFI into a series
of patches, organized as described below (after this one). Please let me
know if you'd like me to split the changes in a different way, or
provide one big patch.
1. The next patch will introduce the `MCLFIExpander` mechanism for
applying the MC-level rewrites needed by LFI, along with the
`.lfi_expand` and `.lfi_no_expand` assembly directives when targeting
LFI. A preview can be seen on the `lfi-project`
[fork](https://github.com/llvm/llvm-project/compare/main...lfi-project:llvm-project:lfi-patchset/aarch64-pr-2).
2. The following patch will create an `MCLFIExpander` for the AArch64
backend that performs LFI expansions. This patch will contain the
majority of the LFI-specific logic.
3. The final patch will add an optimization to the rewriter that can
eliminate redundant guard instructions that occur within the same basic
block.
We plan to introduce x86-64 support after further discussion and once
the `MCLFIExpander` infrastructure is in place.
Please let me know your feedback, and thank you very much for your help
and guidance in the review process.
The existing condition for checking whether or not to expand an frem
instruction in expand-fp is not sufficiently precise.
The expansion on other targets than AMDGPU - which is the only intended
user right now - is only prevented due to the interaction with the
MaxLegalFpConvertBitWidth check. Relying on this is conceptually wrong
and limits the use of the pass for other targets and further expansions
(e.g. merging with the similar ExpandLargeDivRem pass).
Change the expansion criterion to always expand frem of a given type
for targets that use "Expand" as the legalization action for the
underlying scalar type and use this to exit the pass early for targets
which do not require any expansions. This requires to change the
frem legalization action for all targets which do not want frem to
be expanded in this pass from "Expand" to "LibCall".
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Add a mechanism to permit plugins running code between optimizations and
the back-end passes. Implement this through the LLVM plug-in mechanism
to make permit plugins to be written independently of the front-end.
The primary motivation for this point is TPDE-LLVM, which substitutes
the LLVM back-end (optionally falling back to it for unsupported IR). We
have been distributing a Clang patch; but requiring a custom-build
toolchain is impracticable for many users.
Front-end adjustments will follow as separate patches.
Pull Request: https://github.com/llvm/llvm-project/pull/170846
In 4 years the ELF debugger support plugin wasn't adapted to other
object formats or debugging approaches. After the renaming NFC in
https://github.com/llvm/llvm-project/pull/168343, this patch tailors the
plugin to ELF and section load-address patching. It allows removal of
abstractions and consolidate processing steps with the newly enabled
AllocActions from https://github.com/llvm/llvm-project/pull/168343.
The key change is to process debug sections in one place in a
post-allocation pass. Since we can handle the endianness of the ELF file
the single `visitSectionLoadAddresses()` visitor function now, we don't
need to track debug objects and sections in template classes anymore. We
keep using the `DebugObject` class and drop `DebugObjectSection`,
`ELFDebugObjectSection<ELFT>` and `ELFDebugObject`.
Furthermore, we now use the allocation's working memory for load-address
fixups directly. We can drop the `WritableMemoryBuffer` from the debug
object and most of the `finalizeWorkingMemory()` step, which saves one
copy of the entire debug object buffer. Inlining `finalizeAsync()` into
the pre-fixup pass simplifies quite some logic.
The original patch broke compiler-rt checks: db5eeddbd3
Reverted with: 0182a76970
c9821abfc0
added extra fences after sequentially consistent stores for
compatibility with MSVC's seq_cst loads (ldr+dmb). These extra fences
should not be needed for ARM LSE instructions that have both
acquire+release semantics, which results in a two way barrier, and
should be enough for sequential consistency.
Fixes https://github.com/llvm/llvm-project/issues/162345
Change-Id: I9148c73d0dcf3bf1b18a0915f96cac71ac1800f2
This reverts commit 8217c6415ab76c2a0f06705100c76207cd1e6bc0.
I asked Claude Sonnet 4.5 to hazard a guess at what was causing the
buildbot failure, and it seems to have correctly pinned it on a bug in
GCC7, although it didn't give a working fix.
I worked out of a `gcc:7.4` docker container and moved code around until
`MCDwarf.cpp.o` compiled without errors or warnings, so hopefully that's
sufficient to fix the bot.
Adds an option -module-summary-max-indirect-edges, and wiring into the
ICP logic that collects promotion candidates from VP metadata, to
support a larger number of promotion candidates for use in building the
ThinLTO summary. Also use this in the MemProf ThinLTO backend handling
where we perform memprof ICP during cloning.
The new option, essentially off by default, can be used to override the
value of -icp-max-prom, which is checked internally in ICP, with a
larger max value when collecting candidates from the VP metadata.
For MemProf in particular, where we synthesize new VP metadata targets
from allocation contexts, which may not be all that frequent, we need to
be able to include a larger set of these targets in the summary in order
to correctly handle indirect calls in the contexts. Otherwise we will
not set up the callsite graph edges correctly.
This PR adds support for selecting specific archive members in
llvm-symbolizer using the `archive.a(member.o)` syntax, with
architecture-aware member selection.
**Key features:**
1. **Archive member selection syntax**: Specify archive members using
`archive.a(member.o)` format
2. **Architecture selection via `--default-arch` flag**: Select the
appropriate member when multiple members have the same name but
different architectures
3. **Architecture selection via `:arch` suffix**: Alternative syntax
`archive.a(member.o):arch` for specifying architecture
This functionality is primarily designed for AIX big archives, which can
contain multiple members with the same name but different architectures
(32-bit and 64-bit). However, the implementation works with all archive
formats (GNU, BSD, Darwin, big archive) and handles same-named members
created with llvm-ar q.
---------
Co-authored-by: Midhunesh <midhuensh.p@ibm.com>
This change adds support for mixed precision floating point
arithmetic for `f16` and `bf16` where the following patterns:
```
%fh = fpext half %h to float
%resfh = fp-operation(%fh, ...)
...
%fb = fpext bfloat %b to float
%resfb = fp-operation(%fb, ...)
where the fp-operation can be any of:
- fadd
- fsub
- llvm.fma.f32
- llvm.nvvm.add(/fma).*
```
are lowered to the corresponding mixed precision instructions which
combine the conversion and operation into one instruction from
`sm_100` onwards.
This also adds the following intrinsics to complete support for
all variants of the floating point `add/fma` operations in order
to support the corresponding mixed-precision instructions:
- `llvm.nvvm.add.(rn/rz/rm/rp){.ftz}.sat.f`
- `llvm.nvvm.fma.(rn/rz/rm/rp){.ftz}.sat.f`
We lower `fneg` followed by one of the above addition
intrinsics to the corresponding `sub` instruction.
Tests are added in `fp-arith-sat.ll` , `fp-fold-sub.ll`, and
`bultins-nvptx.c`
for the newly added intrinsics and builtins, and in
`mixed-precision-fp.ll`
for the mixed precision instructions.
PTX spec reference for mixed precision instructions:
https://docs.nvidia.com/cuda/parallel-thread-execution/#mixed-precision-floating-point-instructions
This is needed so the llvm-cgdata tool properly builds with
`LLVM_BUILD_LLVM_DYLIB` so LLVM can be built as a DLL on Windows.
This effort is tracked in #109483.
This is the last of the generic instructions created from MVE
intrinsics. It was a little more awkward than the others due to it
taking a Type as one of the arguments. This creates a new function to
create the intrinsic we need.
Following the suggestions in #170061, I replaced `SmallVector<SDValue>`
with `std::array<SDValue, NumPatterns>` and `SmallBitVector` with
`Bitset<NumPatterns>`.
I had to make some changes to the `collectLeaves` and
`reassociatableMatchHelper` functions. In `collectLeaves` specifically,
I changed the return type so I could propagate a failure in case the
number of found leaves is greater than the number of expected patterns.
I also added a new unit test that, together with the one already present
in the previous line, checks if the matching fails in the cases where
the number of patterns is less or more than the number of leaves.
I don't think this is going to completely address the increased compile
time reported in #169644, but hopefully it leads to an improvement.
Considering that the current loop fusion only supports adjacent loops,
we are able to simplify the checks in this pass. By removing
`isControlFlowEquivalent` check, this patch fixes multiple issues
including #166560, #166535, #165031, #80301 and #168263.
Now only the sequential/adjacent candidates are collected in the same
list. This patch is the implementation of approach 2 discussed in post
#171207.
Delinearization has its own `isKnownNonNegative` function, which wraps
`ScalarEvolution::isKnownNonNegative` and adds additional logic. The
additional logic is that, for a pointer addrec `{a,+,b}`, if the pointer
has `inbounds` and both `a` and `b` are known to be non-negative, then
the addrec is also known non-negative (i.e., it doesn't wrap). This
reasoning is incorrect. If the GEP and/or load/store using the pointer
are not unconditionally executed in the loop, then the addrec can still
wrap. Even though no actual example has been found where this causes a
miscompilation (probably because the subsequent checks fail so the
validation also fails), simply replacing it with
`ScalarEvolution::isKnownNonNegative` is safer, especially it doesn't
cause any regressions in the existing tests.
Resolve#169811
Invent `StylizedInstance` class to store special variables together with
the instantiated expression in omp::clause::Initializer. This will
eliminate the need for visiting the original AST nodes in lowering to
MLIR.
