cloneExternalModuleToContext can be used to clone an LLVM module onto a given
ThreadSafeContext. Callers of this function are responsible for ensuring
exclusive access to the source module and its LLVMContext.
This patch fixes:
llvm/unittests/ExecutionEngine/Orc/CoreAPIsTest.cpp:1578:3: error:
'std::unique_lock' may not intend to support class template argument
deduction [-Werror,-Wctad-maybe-unsupported]
The code previously appeared to have a (benign?) race condition on
`WorkThreads.size`, since it was being accessed outside of the mutex
lock that protected it on the threads. This is usually okay since
1a1d6e6f98738be249b20994bcfed48dccac59e3, but doesn't seem reliable in
general, so fix this code to express the intent more accurately. This
instead relies on the same general principles as ref-counting, where
each existing reference (thread) can add new references (threads)
because they already have a reference themselves (until joined).
This is a generalization of the existing cloneToNewContext operation:
rather than cloning the given module into a new ThreadSafeContext it
clones it into any given ThreadSafeContext. The given ThreadSafeContext
is locked to ensure that the cloning operation is safe.
This removes ThreadSafeContext::Lock, ThreadSafeContext::getLock, and
ThreadSafeContext::getContext, and replaces them with a
ThreadSafeContext::withContextDo method (and const override).
The new method can be used to access an existing
ThreadSafeContext-wrapped LLVMContext in a safe way:
ThreadSafeContext TSCtx = ... ;
TSCtx.withContextDo([](LLVMContext *Ctx) {
// this closure has exclusive access to Ctx.
});
The new API enforces correct locking, whereas the old APIs relied on
manual locking (which almost no in-tree code preformed, relying instead
on incidental exclusive access to the ThreadSafeContext).
ArrayRef now has a new constructor that takes a parameter whose type
has data() and size(). This patch migrates:
ArrayRef<T>(X.data(), X.size()
to:
ArrayRef<T>(X)
The write-pointers operation should be writing a pointer, not a uint64_t. This
bug existed prior to f93df5ebd99, but changes in that commit seem to have
exposed the issue (see e.g.
https://lab.llvm.org/buildbot/#/builders/154/builds/17956).
This commit adds operations to orc::MemoryAccess for reading basic types
(uint8_t, uint16_t, uint32_t, uint64_t, pointers, buffers, and strings)
from executor memory.
The InProcessMemoryAccess and EPCGenericMemoryAccess implementations are
updated to support the new operations.
This moves the MemoryAccess interface out of the ExecutorProcessControl
class and splits implementation classes InProcessMemoryManager and
SelfExecutorProcessControl out of ExecutorProcessControl.h and into
their own headers.
`RTDyldObjectLinkingLayer` is currently creating a memory manager
without any parameters.
In this PR I am passing the MemoryBuffer that will be emitted to the
MemoryManager so that the user can use it to configure the behaviour of
the MemoryManager.
## Purpose
This patch is one in a series of code-mods that annotate LLVM’s public
interface for export. This patch annotates the `llvm/ExecutionEngine`
library. These annotations currently have no meaningful impact on the
LLVM build; however, they are a prerequisite to support an LLVM Windows
DLL (shared library) build.
## Background
This effort is tracked in #109483. Additional context is provided in
[this
discourse](https://discourse.llvm.org/t/psa-annotating-llvm-public-interface/85307),
and documentation for `LLVM_ABI` and related annotations is found in the
LLVM repo
[here](https://github.com/llvm/llvm-project/blob/main/llvm/docs/InterfaceExportAnnotations.rst).
The bulk of these changes were generated automatically using the
[Interface Definition Scanner (IDS)](https://github.com/compnerd/ids)
tool, followed formatting with `git clang-format`.
The following manual adjustments were also applied after running IDS on
Linux:
- Add `LLVM_ABI_FRIEND` to friend member functions declared with
`LLVM_ABI`
- Add `LLVM_ABI` to a subset of private class methods and fields that
require export
- Add `LLVM_ABI` to a small number of symbols that require export but
are not declared in headers
- Add `LLVM_ABI` to a number of `extern "C"` methods that IDS missed
because they're implicitly exported
## Validation
Local builds and tests to validate cross-platform compatibility. This
included llvm, clang, and lldb on the following configurations:
- Windows with MSVC
- Windows with Clang
- Linux with GCC
- Linux with Clang
- Darwin with Clang
It failed on armv7 with "Architecture not supported" which is due to
StubTests being not supported on ARM
/builds/fossdd/aports/main/llvm20/src/llvm-project-20.1.0.src/llvm/unittests/ExecutionEngine/Orc/ReOptimizeLayerTest.cpp:140:
Failure
Value of: llvm::detail::TakeError(RM.takeError())
Expected: succeeded
Actual: failed (Architecture not supported) (of type
llvm::detail::ErrorHolder)
David Abdurachmanov reports[1] that we hit this error on P550 hardware:
/builddir/build/BUILD/llvm-20.1.1-build/llvm-project-20.1.1.src/llvm/unittests/ExecutionEngine/Orc/ReOptimizeLayerTest.cpp:140:
Failure
Value of: llvm::detail::TakeError(RM.takeError())
Expected: succeeded
Actual: failed (Architecture not supported) (of type
llvm::detail::ErrorHolder)
Tom Stellard noted[2] that he's seen this before on other architectures
and suggested to skip it.
[1] https://src.fedoraproject.org/rpms/llvm/pull-request/408#comment-255547
[2] https://src.fedoraproject.org/rpms/llvm/pull-request/408#comment-255557
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
This can be used to construct an ExecutorAddrRange from a pair of pointers, or
a pointer and a size.
This will be used to reduce boilerplate in an upcoming patch.
The module currently stores the target triple as a string. This means
that any code that wants to actually use the triple first has to
instantiate a Triple, which is somewhat expensive. The change in #121652
caused a moderate compile-time regression due to this. While it would be
easy enough to work around, I think that architecturally, it makes more
sense to store the parsed Triple in the module, so that it can always be
directly queried.
For this change, I've opted not to add any magic conversions between
std::string and Triple for backwards-compatibilty purses, and instead
write out needed Triple()s or str()s explicitly. This is because I think
a decent number of them should be changed to work on Triple as well, to
avoid unnecessary conversions back and forth.
The only interesting part in this patch is that the default triple is
Triple("") instead of Triple() to preserve existing behavior. The former
defaults to using the ELF object format instead of unknown object
format. We should fix that as well.
Fixes a couple hard coded 16k values which is being used as the page
size. Replaces the hard coded value with the system's page size. This
fixes#116753 on an Ampere Altra Q64-22
CC @lhames
Use SymbolStringPtr for Symbol names in LinkGraph. This reduces string interning
on the boundary between JITLink and ORC, and allows pointer comparisons (rather
than string comparisons) between Symbol names. This should improve the
performance and readability of code that bridges between JITLink and ORC (e.g.
ObjectLinkingLayer and ObjectLinkingLayer::Plugins).
To enable use of SymbolStringPtr a std::shared_ptr<SymbolStringPool> is added to
LinkGraph and threaded through to its construction sites in LLVM and Bolt. All
LinkGraphs that are to have symbol names compared by pointer equality must point
to the same SymbolStringPool instance, which in ORC sessions should be the pool
attached to the ExecutionSession.
---------
Co-authored-by: Lang Hames <lhames@gmail.com>
This reapplies 427fb5cc5ac, which was reverted in 08c1a6b3e18 due to bot
failures.
The fix was to remove an incorrect assertion: In IL_emit, during the initial
worklist loop, an EDU can have all of its dependencies removed without becoming
ready (because it may still have implicit dependencies that will be added back
during the subsequent propagateExtraEmitDeps operation). The EDU will be marked
Ready at the end of IL_emit if its Dependencies set is empty at that point.
Prior to that we can only assert that it's either Emitted or Ready (which is
already covered by other assertions).
AsynchronousSymbolQuery tracks the symbols that it depends on in order to (1)
detach the query in the event of a failure, and (2) report those dependencies
to clients of the ExecutionSession::lookup method (via the RegisterDependencies
argument). Previously we tracked only dependencies on symbols that didn't meet
the required state (the only symbols that the query needs to be attached to),
but this is insufficient to report all necessary dependencies to lookup clients.
E.g. A lookup requiring SymbolState::Resolved where some matched symbol is
already Resolved but not yet Emitted or Ready would result in the dependency on
that symbol not being reported, which could result in illegal access in
concurrent JIT setups. (This bug was discovered by @mikaoP on discord with a
simple concurrent JIT setup).
This patch tracks and reports all dependencies on symbols that aren't Ready yet,
correcting the under-reporting issue. AsynchronousSymbolQuery::detach is updated
to stop asserting that all depended-upon symbols have a query attached.
Make sure the inlined @sum() function has the right extension attributes
on its arguments.
A new struct TargetI32ArgExtensions is added that sets the Ret/Arg extension
strings given a string TargetTriple. This might be used elsewhere as well for
this purpose if needed.
Fixes: #112503
Redirectable stubs should be placed in the same JITDylib as their names, with
their lifetimes managed according to the ResourceTracker used when adding them.
The original implementation created a single pool of stubs in the JITDylib
that is passed to the JITLinkRedirectableSymbolManager constructor, but this
may cause the addresses of the redirectable symbols themselves (added to the
JITDylibs passed to createRedirectableSymbols) to appear outside the address
range of their defining JITDylib.
This patch fixes the issue by dropping the pool and emitting a new graph for
each set of requested redirectable symbols. We lose the ability to recycle
stubs, but gain the ability to free them entirely. Since we don't expect stub
reuse to be a common case this is likely the best trade-off.
If in the future we do need to return to a stub pool we should create a
separate one for each JITDylib to ensure that addresses behave as expected.
The native target must be initialized here otherwise the test will be skipped
unless some prior test happens to initialize it. Failure to initialize the
native target was causing the test to be skipped when --gtest_filter was used.
When transferring resources, the destination tracker key may not be in
the internal map, invalidating iterators and value references. The added
test creates such situation and would fail before with "Finalized
allocation was not deallocated."
For good measure, fix the same pattern in RTDyldObjectLinkingLayer
which is harder to test because it "only" results in memory managers
being deleted in the wrong order.
This keeps common operations together, and should make it easier to write
re-usable dylib managers in the future (e.g. a DylibManager that uses
the EPC's remote-execution APIs to implement load and lookup).
Fix the builds with LLVM_TOOL_LLVM_DRIVER_BUILD enabled.
LLVM_ENABLE_EXPORTED_SYMBOLS_IN_EXECUTABLES is not completely
compatible with export_executable_symbols as the later will be ignored
if the previous is set to NO.
Fix the issue by passing if symbols need to be exported to
llvm_add_exectuable so the link flag can be determined directly
without calling export_executable_symbols_* later.
`LLVM_ENABLE_EXPORTED_SYMBOLS_IN_EXECUTABLES` is not completely
compatible with `export_executable_symbols` as the later will be ignored
if the previous is set to NO.
Fix the issue by passing if symbols need to be exported to
`llvm_add_exectuable` so the link flag can be determined directly
without calling `export_executable_symbols_*` later.
In AllocGroupSmallMap::find(AllocGroup) we were calling lower_bound(...) and
then unconditionally dereferencing the resulting iterator, however
lower_bound(...) may return end() if the value being searched for is higher
than any value present in the map. This patch adds a check for end() before
the dereference to guard against dereference of end().
This commit also adds some basic unit tests for MemProt and AllocGroupSmallMap.
rdar://129662981
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
