Integrate in-memory debug-info dumps into the `--orc-lazy-debug`
command-line option instead of exposing built-in functions to be called
from JITed code. This reduces overall amount of code (removing
`ExecutionUtils.cpp`) and seems cleaner anyway.
All existing items of `OrcDumpKind` work on IR level and run in the
IR-transform step of the JIT. The newly added `DumpDebugDescriptor` and
`DumpDebugObjects` must run after debug-registration and thus are
deferred to the Object-transform step of the JIT. This separation is the
major side-effect of the patch.
The original commit 263efb044add93b9 was reverted in #79055, because
the gcc 7.5 bot had found a missing std::move().
Reverts llvm/llvm-project#76822
This fails to build with gcc 7.5:
```
llvm/tools/lli/lli.cpp:1087:16: error: could not convert ‘Obj’ from ‘std::unique_ptr<llvm::MemoryBuffer>’ to ‘llvm::Expected<std::unique_ptr<llvm::MemoryBuffer> >’
return Obj;
^~~
```
Integrate in-memory debug-info dumps into the `--orc-lazy-debug`
command-line option instead of exposing built-in functions to be called
from JITed code. This reduces overall amount of code (removing
`ExecutionUtils.cpp`) and seems cleaner anyway.
All existing items of `OrcDumpKind` work on IR level and run in the
IR-transform step of the JIT. The newly added `DumpDebugDescriptor` and
`DumpDebugObjects` must run after debug-registration and thus are
deferred to the Object-transform step of the JIT. This separation is the
major side-effect of the patch.
So that when mixing small and large text, large text stays out of the
way of the rest of the binary.
This is useful for mixing precompiled small code model object files and
built-from-source large code model binaries so that the the text
sections don't get merged.
The reland fixes an issue where a function in the large code model would reference small data without GOTOFF.
This was incorrectly reverted in 76f78ecc789d58baa3a88b2fe2a57428f07e5362.
This reverts commit 4bf8a688956a759b7b6b8d94f42d25c13c7af130.
This commit seems to be breaking the semantics of the
ObjectFile::isSectionText method, which breaks numba/llvmlite bindings.
So that when mixing small and large text, large text stays out of the
way of the rest of the binary.
This is useful for mixing precompiled small code model object files and
built-from-source large code model binaries so that the the text
sections don't get merged.
The reland fixes an issue where a function in the large code model would reference small data without GOTOFF.
So that when mixing small and large text, large text stays out of the
way of the rest of the binary.
This is useful for mixing precompiled small code model object files and
built-from-source large code model binaries so that the the text
sections don't get merged.
This is a follow-up to b71edfaa4ec3c998aadb35255ce2f60bba2940b0
since I forgot the lit.local.cfg files in that one.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: barannikov88, kwk
Differential Revision: https://reviews.llvm.org/D150762
Prior to this patch, for the DWARF frame base LLVM uses the frame pointer
register if available, otherwise the stack pointer register. If the stack
pointer register is being used and a call or other code modifies the stack
pointer during the body of the function this results in the locations being
wrong and the debugger displaying the wrong values for variables.
By using DW_OP_call_frame_cfa in these situations the emitted location for
the variable will automatically handle changes in the stack pointer.
The CFA needs to be adjusted for the offset between the frame pointer/stack
pointer to allow the variable locations themselves to remain unchanged by
this patch.
Reviewed By: #debug-info, scott.linder, jryans
Differential Revision: https://reviews.llvm.org/D143463
This reapplies 371cb1af61d, which was reverted in 0b2240eda01 due to bot
failures.
The clang-repl test failure is fixed by dropping the process symbols definition
generator that was manually attached to the main JITDylib, since LLJIT now
exposes process symbols by default. (The bug was triggered when JIT'd code used
the process atexit provided by the generator, rather than the JIT atexit which
has been moved into the platform JITDylib).
Any LLJIT clients that see crashes in static destructors should likewise remove
any process symbol generators attached to their main JITDylib.
This commit includes several related ergonomic improvements to LLJIT.
(1) Adds a default JITDylibSearchOrder to be appended to the initial link order
of JITDylibs created via LLJIT::createJITDylib (dropping any duplicate entries).
This was introduced to support automatic reflection of process symbols (see
(2) below), but has been made visible to clients as it's generically useful,
e.g. if clients have some extra set of libraries that they want to be visible
to JIT'd code by default.
The default JITDylibSearchOrder is only appended to the link order of JITDylibs
created via LLJIT::createJITDylib, and will not be apply to JITDylibs created by
directly calling the underlying ExecutionSession -- in that case clients can set
up the link order manually.
(2) Makes process symbols visible to JIT'd code by default via the new "Process"
JITDylib, which is added to the default link order.
LLJIT clients usually want symbols in the executor process to be accessible to
JIT'd code. Until now clients have been left to set this up themselves by adding
a DynamicLibrarySearchGenerator to the Main JITDylib. This patch adds a new
process symbols JITDylib that will be created by default (with an
EPCDynamicLibrarySearchGenerator attached) and added to the default link order,
making process symbols available to JIT'd code.
Clients who do not want process symbols to be visible to JIT'd code by default
can call setLinkProcessSymbolsByDefault(false) on their LLJITBuilder to disable
this:
LLJITBuilder()
...
.setLinkProcessSymbolsByDefault(false)
...
.create();
Clients can also call setProcessSymbolsJITDylibSetup to take over responsibility
for configuring the process symbols JITDylib (the callback that the client
supplies will be called on the bare process symbols JITDylib immediately after
it is created).
If setLinkProcessSymbolsByDefault(false) is called and no JITDylib setup
callback has been set then the process symbols JITDylib will not be created and
LLJIT::getProcessSymbolsJITDylib will return null.
(3) Adds an ExecutorNativePlatform utility that makes it easier to enable
native platform features.
Some object format features (e.g. native static initializers and thread locals)
require runtime support in the executing process. Support for these features in
ORC is implemented cooperatively between the ORC runtime and the LLVM Platform
subclasses (COFFPlatform, ELFNixPlatform, and MachOPlatform).
ExecutorNativePlatfrom simplifies the process of loading the ORC runtime and
creating the appropriate platform class for the executor process.
ExecutorNativePlatform takes a path to the ORC runtime (or a MemoryBuffer
containing the runtime) and other required runtimes for the executor platform
(e.g. MSVC on Windows) and then configures LLJIT with an appropriate platform
class based on the executor's target triple:
LLJITBuilder()
.setPlatformSetUp(ExecutorNativePlatform("/path/to/orc-runtime.a"));
(The ORC runtime is built as part of compiler-rt, and the exact name of the
archive is platform dependent).
The ORC runtime and platform symbols will be added to a new "Platform" JITDylib,
which will be added to the *front* of the default link order (so JIT'd code will
prefer symbol definitions in the platform/runtime to definitions in the executor
process).
ExecutorNativePlatform assumes that the Process JITDylib is available, as
the ORC runtime may depend on symbols provided by the executor process.
Differential Revision: https://reviews.llvm.org/D144276
This reverts commit bdf5f9c3228d6ed1d7c6f87b3828a7d573b34c03, which was a work
in progress for https://reviews.llvm.org/D144276 and accidentally committed
early.
Simplifies the process of building an LLJIT instance that supports the native
platform features (initializers, TLV, etc.).
SetUpExecutorNativePlatform can be passed to LLJITBuilder::setPlatformSetUp
method. It takes a reference to the ORC runtime (as a path or an in-memory
archive) and automatically sets the platform for LLJIT's ExecutionSession based
on the executor process's triple.
Differential Revision: https://reviews.llvm.org/D144276
This patch also sets `UNSUPPORTED` on some tests which need `mcjit` and
`emulated tls` support.
Depends on D142950
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D142954
Define atexit symbol in GenericLLVMIRPlatformSupport so that it doesn't need to be defined by user.
On windows, llvm codegen emits atexit runtime calls to support global deinitializers as there is no lower function like cxa_atexit as in Itanium C++ ABI. ORC JIT user had to define custom atexit symbol manually. This was a hassle as it has to deal with dso_handle and cxa_atexit internals of LLJIT. If client didn't provide atexit definition, the default behaviour is just linking with host atexit function which is destined to fail as it calls dtors when the host program exits. This is after jit instances and buffers are freed, so users would see weird access violation exception from the uknown location. (in console application, the debugger thinks exception happened in scrt_common_main_seh)
This is a hack that has some caveats. (e.g. memory address is not identical) But, it's better than the situation described in the above. Ultimately, we will move on to ORC runtime that is able to solve the memory address issue properly.
Reviewed By: sgraenitz
Differential Revision: https://reviews.llvm.org/D128037
Initially, lli only supported lazy mode for ORC. Greedy mode was added with e1579894d205 and it's the default setting now. DebugObjectManagerPlugin tests don't rely on laziness, so we can switch them to greedy in order to avoid some unnecessary complexity.
These tests were disabled by accident after D107640. Actually, REQUIRES lines don't support `x86_64` and so these tests stopped running on all targets.
`native && target-x86_64` should be the correct term to express "x86_64 host targeting native arch".
These tests rely on running IR code with an explicit x86_64 target triple. They won't work on other architectures. (They won't work for 32-bit processes on x86_64 hosts either. We will take care of this later.)
Differential Revision: https://reviews.llvm.org/D107640
When lli runs the below IR, it emits in-memory debug objects and registers them with the GDB JIT interface. The tests dump and check the registered information. IR has limited ability to produce complex output in a portable way. Instead the tests rely on built-in functions implemented in lli. They use a new command line flag `-generate=function-name` to instruct the ORC JIT to expose the built-in function with the given name to the JITed program.
`debug-descriptor-elf-minimal.ll` calls `__dump_jit_debug_descriptor()` to reflect the list of debug entries issued for itself after emitting the main module. The output is textual and can be checked straight away.
`debug-objects-elf-minimal.ll` calls `__dump_jit_debug_objects()`, which instructs lli to walk through the list of debug entries and append the encountered in-memory objects to the program output. We feed this output into llvm-dwarfdump to parse the DWARF in each file and dump their structures.
We can do the same for JITLink once D97335 has landed.
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D97694
If a symbol name begins with the linker private global prefix (as
described by the DataLayout) then it should be treated as non-exported,
regardless of its LLVM IR visibility value.
Add a new overload of StaticLibraryDefinitionGenerator::Load that takes a triple
argument and supports loading archives from MachO universal binaries in addition
to regular archives.
The LLI tool is updated to use this overload.
Failure to export __cxa_atexit can lead to an attempt to import a definition
from the process itself (if __cxa_atexit is referenced from another JITDylib),
but the process definition will clash with the existing non-exported definition
to produce an unexpected DuplicateDefinitionError.
This patch fixes the immediate issue by exporting __cxa_atexit. It also fixes a
bug where atexit functions in other JITDylibs were not being run by adding a
copy of run_atexits_helper to every JITDylib.
A follow up patch will deal with the bug where definition generators are called
despite a non-exported definition being present.
The MemoryBuffer::getMemBuffer method's RequiresNullTerminator parameter
defaults to true, but object files are not null terminated so we need to
explicitly pass false here.
This patch enables exception handling in code added to LLJIT on Darwin by
adding an orc::EHFrameRegistrationPlugin instance to the ObjectLinkingLayer
(which is currently used on Darwin only).
The GenericLLVMIRPlatformSupport class runs a transform on all LLVM IR added to
the LLJIT instance to replace instances of llvm.global_ctors with a specially
named function that runs the corresponing static initializers (See
(GlobalCtorDtorScraper from lib/ExecutionEngine/Orc/LLJIT.cpp). This patch
updates the GenericIRPlatform class to check for this specially named function
in other materialization units that are added to the JIT and, if found, add
the function to the initializer work queue. Doing this allows object files
that were compiled from IR and cached to be reloaded in subsequent JIT sessions
without their initializers being skipped.
To enable testing this patch also updates the lli tool's -jit-kind=orc-lazy mode
to respect the -enable-cache-manager and -object-cache-dir options, and modifies
the CompileOnDemandLayer to rename extracted submodules to include a hash of the
names of their symbol definitions. This allows a simple object caching scheme
based on module names (which was already implemented in lli) to work with the
lazy JIT.
Initializers and deinitializers are used to implement C++ static constructors
and destructors, runtime registration for some languages (e.g. with the
Objective-C runtime for Objective-C/C++ code) and other tasks that would
typically be performed when a shared-object/dylib is loaded or unloaded by a
statically compiled program.
MCJIT and ORC have historically provided limited support for discovering and
running initializers/deinitializers by scanning the llvm.global_ctors and
llvm.global_dtors variables and recording the functions to be run. This approach
suffers from several drawbacks: (1) It only works for IR inputs, not for object
files (including cached JIT'd objects). (2) It only works for initializers
described by llvm.global_ctors and llvm.global_dtors, however not all
initializers are described in this way (Objective-C, for example, describes
initializers via specially named metadata sections). (3) To make the
initializer/deinitializer functions described by llvm.global_ctors and
llvm.global_dtors searchable they must be promoted to extern linkage, polluting
the JIT symbol table (extra care must be taken to ensure this promotion does
not result in symbol name clashes).
This patch introduces several interdependent changes to ORCv2 to support the
construction of new initialization schemes, and includes an implementation of a
backwards-compatible llvm.global_ctor/llvm.global_dtor scanning scheme, and a
MachO specific scheme that handles Objective-C runtime registration (if the
Objective-C runtime is available) enabling execution of LLVM IR compiled from
Objective-C and Swift.
The major changes included in this patch are:
(1) The MaterializationUnit and MaterializationResponsibility classes are
extended to describe an optional "initializer" symbol for the module (see the
getInitializerSymbol method on each class). The presence or absence of this
symbol indicates whether the module contains any initializers or
deinitializers. The initializer symbol otherwise behaves like any other:
searching for it triggers materialization.
(2) A new Platform interface is introduced in llvm/ExecutionEngine/Orc/Core.h
which provides the following callback interface:
- Error setupJITDylib(JITDylib &JD): Can be used to install standard symbols
in JITDylibs upon creation. E.g. __dso_handle.
- Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU): Generally
used to record initializer symbols.
- Error notifyRemoving(JITDylib &JD, VModuleKey K): Used to notify a platform
that a module is being removed.
Platform implementations can use these callbacks to track outstanding
initializers and implement a platform-specific approach for executing them. For
example, the MachOPlatform installs a plugin in the JIT linker to scan for both
__mod_inits sections (for C++ static constructors) and ObjC metadata sections.
If discovered, these are processed in the usual platform order: Objective-C
registration is carried out first, then static initializers are executed,
ensuring that calls to Objective-C from static initializers will be safe.
This patch updates LLJIT to use the new scheme for initialization. Two
LLJIT::PlatformSupport classes are implemented: A GenericIR platform and a MachO
platform. The GenericIR platform implements a modified version of the previous
llvm.global-ctor scraping scheme to provide support for Windows and
Linux. LLJIT's MachO platform uses the MachOPlatform class to provide MachO
specific initialization as described above.
Reviewers: sgraenitz, dblaikie
Subscribers: mgorny, hiraditya, mgrang, ributzka, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D74300
This commit adds a ManglingOptions struct to IRMaterializationUnit, and replaces
IRCompileLayer::CompileFunction with a new IRCompileLayer::IRCompiler class. The
ManglingOptions struct defines the emulated-TLS state (via a bool member,
EmulatedTLS, which is true if emulated-TLS is enabled and false otherwise). The
IRCompileLayer::IRCompiler class wraps an IRCompiler (the same way that the
CompileFunction typedef used to), but adds a method to return the
IRCompileLayer::ManglingOptions that the compiler will use.
These changes allow us to correctly determine the symbols that will be produced
when a thread local global variable defined at the IR level is compiled with or
without emulated TLS. This is required for ORCv2, where MaterializationUnits
must declare their interface up-front.
Most ORCv2 clients should not require any changes. Clients writing custom IR
compilers will need to wrap their compiler in an IRCompileLayer::IRCompiler,
rather than an IRCompileLayer::CompileFunction, however this should be a
straightforward change (see modifications to CompileUtils.* in this patch for an
example).
This fixes an off-by-one error in the argc value computed by runAsMain, and
switches lli back to using the input bitcode (rather than the string "lli") as
the effective program name.
Thanks to Stefan Graenitz for spotting the bug.
This patch replaces the JITDylib::DefinitionGenerator typedef with a class of
the same name, and adds support for attaching a sequence of DefinitionGeneration
objects to a JITDylib.
This patch also adds a new definition generator,
StaticLibraryDefinitionGenerator, that can be used to add symbols fom a static
library to a JITDylib. An object from the static library will be added (via
a supplied ObjectLayer reference) whenever a symbol from that object is
referenced.
To enable testing, lli is updated to add support for the --extra-archive option
when running in -jit-kind=orc-lazy mode.
llvm-svn: 368707
Doesn't build on Windows. The call to 'lookup' is ambiguous. Clang and
MSVC agree, anyway.
http://lab.llvm.org:8011/builders/clang-x64-windows-msvc/builds/787
C:\b\slave\clang-x64-windows-msvc\build\llvm.src\unittests\ExecutionEngine\Orc\CoreAPIsTest.cpp(315): error C2668: 'llvm::orc::ExecutionSession::lookup': ambiguous call to overloaded function
C:\b\slave\clang-x64-windows-msvc\build\llvm.src\include\llvm/ExecutionEngine/Orc/Core.h(823): note: could be 'llvm::Expected<llvm::JITEvaluatedSymbol> llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib *>,llvm::orc::SymbolStringPtr)'
C:\b\slave\clang-x64-windows-msvc\build\llvm.src\include\llvm/ExecutionEngine/Orc/Core.h(817): note: or 'llvm::Expected<llvm::JITEvaluatedSymbol> llvm::orc::ExecutionSession::lookup(const llvm::orc::JITDylibSearchList &,llvm::orc::SymbolStringPtr)'
C:\b\slave\clang-x64-windows-msvc\build\llvm.src\unittests\ExecutionEngine\Orc\CoreAPIsTest.cpp(315): note: while trying to match the argument list '(initializer list, llvm::orc::SymbolStringPtr)'
llvm-svn: 345078
In the new scheme the client passes a list of (JITDylib&, bool) pairs, rather
than a list of JITDylibs. For each JITDylib the boolean indicates whether or not
to match against non-exported symbols (true means that they should be found,
false means that they should not). The MatchNonExportedInJD and MatchNonExported
parameters on lookup are removed.
The new scheme is more flexible, and easier to understand.
This patch also updates JITDylib search orders to be lists of (JITDylib&, bool)
pairs to match the new lookup scheme. Error handling is also plumbed through
the LLJIT class to allow regression tests to fail predictably when a lookup from
a lazy call-through fails.
llvm-svn: 345077
