This PR reapplies the changes previously introduced in
https://github.com/llvm/llvm-project/pull/148410.
It introduces a redesigned and rebuilt Cling-based auto-loading
workaround that enables scanning libraries and resolving unresolved
symbols within those libraries.
Fix build failures in LibraryResolverTest and silence symlink warning
This commit resolves issues observed in the build bots:
1. Silences the -Wunused-result warning by handling the return value
of ::symlink in LibraryResolverTest.cpp. Previously, ignoring
the return value triggered compiler warnings.
2. Fixes a linker error in OrcJITTests caused by an undefined
symbol: llvm::yaml::convertYAML. The test setup in
LibraryResolverTest.cpp now correctly links against the required
LLVM YAML library symbols.
3. Fixes persistent build bot failure caused by a path difference issue.
This resolves the build failures for PR
https://github.com/llvm/llvm-project/pull/148410 on the affected bots.
After #164340 there is a tsan race on `OutstandingSymbolsCount` when
decrementing it in `notifySymbolMetRequiredState` vs reading it in
`isComplete()`. Fix this by having `IL_emit` filter out non-completed
queries when it has the lock to do so, and that way we avoid needing to
call `isComplete()` later.
This PR reapplies the changes previously introduced in #148410.
It introduces a redesigned and rebuilt Cling-based auto-loading
workaround that enables scanning libraries and resolving unresolved
symbols within those libraries.
…fixes.
This reapplies c8c86efbbb5, which was reverted in 13ca8723d1b due to bot
failures. Those failures were compilation errors exposed when LLVM is
built with LLVM_ENABLE_EXPENSIVE_CHECKS=On. They have been fixed in this
commit.
WaitingOnGraph tracks waiting-on relationships between nodes (intended
to represent symbols in an ORC program) in order to identify nodes that
are *Ready* (i.e. are not waiting on any other nodes) or have *Failed*
(are waiting on some node that cannot be produced).
WaitingOnGraph replaces ORC's baked-in data structures that were
tracking the same information (EmissionDepUnit, EmissionDepUnitInfo,
...). Isolating this information in a separate data structure simplifies
the code, allows us to unit test it, and simplifies performance testing.
The WaitingOnGraph uses several techniques to improve performance
relative to the old data structures, including symbol coalescing
("SuperNodes") and symbol keys that don't perform unnecessary reference
counting (NonOwningSymbolStringPtr).
This commit includes unit tests for common dependence-tracking issues
that have led to ORC bugs in the past.
This PR introduces a redesigned and rebuilt Cling-based auto-loading
workaround, which enables scanning libraries and searching for
unresolved symbols within the scanned libraries.
SimpleRemoteMemoryMapper is a MemoryMapper implementation that manages
remote memory via EPC-calls to reserve, initialize, deinitialize, and
release operations. It is compatible with the
SimpleExecutorMemoryManager backend, and its introduction allows
MapperJITLinkMemoryManager to use this backend.
It is also intended to be compatible with the
orc_rt::SimpleNativeMemoryMap backend.
Teach ExecutorSimpleMemoryManager to handle slab reserve/release
operations, plus separate initialize/deinitialize for regions within the
slab. The release operation automatically deinitializes any regions
within each slab that have not already been released.
EPCGenericJITLinkMemoryManager is updated to use the reserve (allocate),
initialize (finalize), and relesae (deallocate) operations.
This brings ExecutorSimpleMemoryManager into alignment with the
orc_rt::SimpleNativeMemoryMap class, allowing SimpleNativeMemoryMap to
be used as a backend for EPCGenericJITLinkMemoryManager.
A future commit will introduce a new MemoryMapper class that will make
SimpleNativeMemoryMap usable as a backend for
MapperJITLinkMemoryManager.
This work will make it easier to re-use in-tree APIs and tools with the
new ORC runtime.
This reverts commit 3b5842c9c41a441280100045ef62bb8a0fe7200f.
The intent of the original commit was to begin enabling asynchronous
alloation actions (calls attached to JIT'd memory initialization and
deinitialization). The asynchronous allocation actions scheme was
fleshed-out in a development branch, but ran into an issue: Functions
implementing actions are allowed to live in JIT'd code (e.g. in the ORC
runtime), but we can't genally rely on tail-call elimination kicking in.
This resulting in dealloc actions returning via stack frames that had
been deallocated, triggering segfaults.
It's possible that there are other approaches that would allow
asynchronous allocation actions to work, but they're not on the critical
path for JIT improvements so for now we'll just revert.
With the help of @lhames, This pull request introduces the dlupdate
function in the ORC runtime. dlupdate enables incremental execution of
new initializers introduced in the REPL environment. Unlike traditional
dlopen, which manages initializers, code mapping, and library reference
counts, dlupdate focuses exclusively on running new initializers.
This patch adds the new **executor-side resolver API** as suggested by
@lhames. It introduces a `DylibSymbolResolver` that helps resolve
symbols for each loaded dylib.
Previously, we returned a `DylibHandle` to the controller. Now, we wrap
the native handle inside `DylibSymbolResolver` and return a
`ResolverHandle` instead. This makes the code cleaner and separates the
symbol resolution logic from raw handle management.
Adds the name and triple of the graph to LinkGraph::dump output before
the rest of the graph content. Calls from JITLinkGeneric.cpp to dump the
graph are updated to avoid redundantly naming the graph.
We just replaced SmallSet<T *, N> with SmallPtrSet<T *, N>, bypassing
the redirection found in SmallSet.h. With that, we no longer need to
include SmallSet.h in many files.
This patch replaces SmallSet<T *, N> with SmallPtrSet<T *, N>. Note
that SmallSet.h "redirects" SmallSet to SmallPtrSet for pointer
element types:
template <typename PointeeType, unsigned N>
class SmallSet<PointeeType*, N> : public SmallPtrSet<PointeeType*, N>
{};
We only have 140 instances that rely on this "redirection", with the
vast majority of them under llvm/. Since relying on the redirection
doesn't improve readability, this patch replaces SmallSet with
SmallPtrSet for pointer element types.
cloneExternalModuleToContext can be used to clone an LLVM module onto a given
ThreadSafeContext. Callers of this function are responsible for ensuring
exclusive access to the source module and its LLVMContext.
They are already implicitly convertible to the underlying type, but that
doesn't work in some contexts, and it can be useful to get the
underlying value without needing the remember/guess the right type.
I converted a couple of call sites to demonstrate usefulness, but
there's likely more of them. I know at least of at least a few in LLDB,
but I don't want to make this a cross-project patch.
Previously, `result` was checked before handling the `Error` returned by
`callSPSWrapper`. If `Error` was `success` but `result` indicated
failure, the original `Error` was silently dropped, triggering an
assertion failure for unhandled `Error`.
This patch ensures the `Error` is checked and returned before inspecting
`result`, preventing assertion failures.
Co-authored-by: Anutosh Bhat <andersonbhat491@gmail.com>
This is a generalization of the existing cloneToNewContext operation:
rather than cloning the given module into a new ThreadSafeContext it
clones it into any given ThreadSafeContext. The given ThreadSafeContext
is locked to ensure that the cloning operation is safe.
This removes ThreadSafeContext::Lock, ThreadSafeContext::getLock, and
ThreadSafeContext::getContext, and replaces them with a
ThreadSafeContext::withContextDo method (and const override).
The new method can be used to access an existing
ThreadSafeContext-wrapped LLVMContext in a safe way:
ThreadSafeContext TSCtx = ... ;
TSCtx.withContextDo([](LLVMContext *Ctx) {
// this closure has exclusive access to Ctx.
});
The new API enforces correct locking, whereas the old APIs relied on
manual locking (which almost no in-tree code preformed, relying instead
on incidental exclusive access to the ThreadSafeContext).
ArrayRef now has a new constructor that takes a parameter whose type
has data() and size(). This patch migrates:
ArrayRef<T>(X.data(), X.size()
to:
ArrayRef<T>(X)
This commit adds operations to orc::MemoryAccess for reading basic types
(uint8_t, uint16_t, uint32_t, uint64_t, pointers, buffers, and strings)
from executor memory.
The InProcessMemoryAccess and EPCGenericMemoryAccess implementations are
updated to support the new operations.
This moves the MemoryAccess interface out of the ExecutorProcessControl
class and splits implementation classes InProcessMemoryManager and
SelfExecutorProcessControl out of ExecutorProcessControl.h and into
their own headers.
When JITLink reports an out-of-range error, the underlying reason could
be hidden from the user if it's due to an excessively large target
addend. Add non-zero target addend to the message for clarity.
LookupKind::DLSym should only be used for lookups issued on behalf of the ORC
runtime's emulated dlsym operation.
This should fix a bug where ORC runtime clients are unable to access functions
exported by the runtime.
https://github.com/llvm/llvm-project/issues/145296
