I'm in the process of making `LoadScriptingResources` interactively ask
a user whether to load a script. I'd like to turn the existing warning
into the prompt. The simplest way to achieve this is to not print into a
`feedback_stream` parameter, and instead create a prompt right there.
This patch removes the `feedback_stream` parameter and emits a
`ReportWarning` instead. If we get around to adding the prompt instead
of the warning, those changes will be simpler to review. But even if we
don't end up replacing the warning with a prompt, moving away from
output parameters and towards more structured error reporting is a
nice-to-have (e.g., the `warning` prefix is now colored, IDEs have more
flexibility on how to present the warning, etc.).
For a command-line user nothing should change with this patch (apart
from `warning:` being highlighted).
This makes the constructions of string errors more concise and more
consistent, mainly by removing the `inconvertibleErrorCode()`.
Additional changes replace `createStringError(formatv(...), ...)` with
`createStringErrorV(...)`.
Assisted-by: Claude
After cleaning up some of our `LanguageType`/`SourceLangage`
round-tripping (see `7f51a2a47d2e706d04855b0e41690ebafa2b3238`), a CU
with `DW_LANG_MIPS_Assembler` will get a language type of
`eLanguageTypeAssembly` (as opposed to `eLanguageTypeMipsAssembler`).
Reason being that there is no `DW_LNAME_` (DWARFv6 language code) for
`MIPS Assembler`, only for generic `Assembly`. So it's not possible to
round-trip cleanly between pre-DWARFv6 and DWARFv6 language codes, which
LLDB relies on for storing language types (and will lean into more
heavily in the future). This broke a special provision we have where we
allow `ScratchTypeSystemClang` to be used when evaluating expressions in
assembly CUs (i.e., CUs where the debug-info explicitly sets the
language to assembly).
If we ever want to distinguish MIPS from other Assembly, the proper way
to do so is introduce a `DW_LNAME_Mips_Assembler`. For now, this patch
adds another case for `eLanguageTypeAssembly` in
`GetScratchTypeSystemForLanguage`.
The test is a bit quirky because it checks for error messages in all
cases. For the `TypeSystem`s to exist we would have to initialize the
`TypeSystem` plugins, which I couldn't find a precedent for. But happy
to hear other suggestions for testing
Fixes https://github.com/llvm/llvm-project/issues/183388
This revert #181334 and its follow-up PRs (including #181488, #181492,
#181493, #181494 and #181498) as well as Ismail's documentation changes
(#181594, #181717). The original commit causes a test failure in CI
(https://github.com/llvm/llvm-project/issues/181938) but the more I look
at the patch, the more I'm convinced it was not ready to land. It will
be easier to iterate on the feedback by re-landing this than by using
post-commit review.
## Summary
- Move `GetScriptedSymbolLocatorClassName()` from inline in `Target.h`
to out-of-line in `Target.cpp` to avoid collision with Windows
`winuser.h` `#define GetClassName GetClassNameW` macro.
- Replace `LaunchSimple(None, None, os.getcwd())` with
`lldbutil.run_to_breakpoint_do_run()` in `test_locate_source_file` to
fix test failure on remote platforms where the local working directory
doesn't exist.
Fixes CI failures from #181334.
## Test plan
- [ ] Windows (aarch64-windows) build passes
- [ ] remote-linux-win test passes
- [ ] Existing ScriptedSymbolLocator tests pass on local platforms
Co-authored-by: Rahul Reddy Chamala <rachamal@meta.com>
## Summary
Based on discussion from
[RFC](https://discourse.llvm.org/t/rfc-python-callback-for-source-file-resolution/83545),
this PR adds a new `SymbolLocatorScripted` plugin that allows Python
scripts to implement custom symbol and source file resolution logic.
This enables downstream users to build custom symbol servers, source
file remapping, and build artifact resolution entirely in Python.
### Changes
- Adds `LocateSourceFile()` to the SymbolLocator plugin interface,
called during source path resolution with a fully loaded `ModuleSP`, so
the plugin has access to the module's UUID, file paths, and symbols.
- Adds `SymbolLocatorScripted` plugin that delegates all four
SymbolLocator methods (`LocateExecutableObjectFile`,
`LocateExecutableSymbolFile`, `DownloadObjectAndSymbolFile`,
`LocateSourceFile`) to a user-provided Python class.
- Adds `ScriptedSymbolLocatorPythonInterface` to bridge C++ calls to
Python, with proper GIL management and error handling.
- Results for `LocateSourceFile` are cached per (module UUID, source
file) pair.
- The Python class is configured via: `settings set
plugin.symbol-locator.scripted.script-class module.ClassName`
### Python class interface
```python
class MyLocator:
def __init__(self, exe_ctx, args): ...
def locate_source_file(self, module, original_source_file):
...
def locate_executable_object_file(self, module_spec): ...
def locate_executable_symbol_file(self, module_spec,
default_search_paths): ...
def download_object_and_symbol_file(self, module_spec,
force_lookup, copy_executable): ...
```
### Test plan
```
Added TestScriptedSymbolLocator.py with 3 test cases:
- test_locate_source_file — verifies the locator resolves source
files, receives a valid SBModule with UUID, and remaps paths correctly
- test_locate_source_file_none_fallthrough — verifies returning
None falls through to default LLDB resolution, and that having no script
class set works normally
- test_invalid_script_class — verifies graceful handling of
invalid class names without crashing
```
Co-authored-by: Rahul Reddy Chamala <rachamal@fb.com>
In #168245, I attempted to dump the available settings to Markdown. That
required a full build of LLDB. However, to build the docs, only the swig
wrappers should need to be compiled. The comment was that we should be
able to use the definitions from the TableGen files.
Currently, the property definitions in don't have information about the
path where they will be available. They only contain a `Definition`
which groups properties, so they can be added to
`OptionValueProperties`.
With this PR, I'm adding the path for each property definition. For
example, `symbols.enable-external-lookup` would have `Name =
enable-external-lookup, Path = symbols`. In LLDB itself, we don't need
this path, we only need it for the documentation. To avoid mismatches
between the actual path and the declared one, I added a debug-only check
when a property group is added to a parent
(`OptionValueProperties::AppendProperty`).
The TableGen emitter for the properties now additionally emits
`g_{definition}_properties_def`, which includes both the array of
properties and the expected path. This constant has to be used to
initialize a `OptionValueProperties`.
I couldn't test this for everything (e.g. IntelPT or ProcessKDP), but
the necessary changes are simple: (1) set the `Path` in the TableGen
file, (2) update `initialize` to use `_def`.
We want to reload the call stack whenever the frame providers are
updated. To do so, we now emit a `eBroadcastBitStackChanged` on all
threads whenever any changes to the frame providers take place.
I found this very useful while iterating on a frame provider in
lldb-dap. So far, the new frame provider only took effect after
continuing execution. Now the backtrace in VS-Code gets refreshed
immediately upon running `target frame-provider add`.
The motivation here is that we don't want to pollute the SBAPI with
getters/setters for expression evaluation options that only apply to a
single language. The ultimate goal would be to have plugins register
additional options to the `expression` command when the plugin is
loaded. This patch only provides the minimal `SBExpressionOptions`
interface to set an option with an arbitrary name, which the language
plugin knows how to interpret. The underlying options dictionary is an
`StructuredData::Dictionary` so we can map strings to values of any
type. But the SBAPI just exposes setting a boolean value. Future
overloads of `SetLanguageOption` can provide setters for more types.
The boolean setter/getter will be used for the C++-specific option being
introduced in: https://github.com/llvm/llvm-project/pull/177926
This method puts strings into the ConstString pool and vends them as
llvm::StringRefs. Most of the uses only require a `std::string` or a
`const char *`. This can be achieved without wasting memory.
Both `Target::ReadSignedIntegerFromMemory()` and
`Process::ReadSignedIntegerFromMemory()` internally created an unsigned
scalar, so extending the value later did not duplicate the sign bit.
## Summary:
This change introduces a `DAPSessionManager` to enable multiple DAP
sessions to share debugger instances when needed, for things like child
process debugging and some scripting hooks that create dynamically new
targets.
Changes include:
- Add `DAPSessionManager` singleton to track and coordinate all active DAP
sessions
- Support attaching to an existing target via its globally unique target
ID (targetId parameter)
- Share debugger instances across sessions when new targets are created
dynamically
- Refactor event thread management to allow sharing event threads
between sessions and move event thread and event thread handlers to `EventHelpers`
- Add `eBroadcastBitNewTargetCreated` event to notify when new targets are
created
- Extract session names from target creation events
- Defer debugger initialization from 'initialize' request to
'launch'/'attach' requests. The only time the debugger is used currently
in between its creation in `InitializeRequestHandler` and the `Launch`
or `Attach` requests is during the `TelemetryDispatcher` destruction
call at the end of the `DAP::HandleObject` call, so this is safe.
This enables scenarios when new targets are created dynamically so that
the debug adapter can automatically start a new debug session for the
spawned target while sharing the debugger instance.
## Tests:
The refactoring maintains backward compatibility. All existing DAP test
cases pass.
Also added a few basic unit tests for DAPSessionManager
```
>> ninja DAPTests
>> ./tools/lldb/unittests/DAP/DAPTests
>>./bin/llvm-lit -v ../llvm-project/lldb/test/API/tools/lldb-dap/
```
In C++17, static constexpr members are implicitly inline, so they no
longer require an out-of-line definition.
Identified with readability-redundant-declaration.
Another attempt at resolving the deadlock issue @GeorgeHuyubo discovered
(his previous
[attempt](https://github.com/llvm/llvm-project/pull/160225)).
This change can be summarized as the following:
* Plumb through a boolean flag to force no preload in
`GetOrCreateModules` all the way through to `LoadModuleAtAddress`.
* Parallelize `Module::PreloadSymbols` separately from
`Target::GetOrCreateModule` and its caller `LoadModuleAtAddress` (this
is what avoids the deadlock).
These changes roughly maintain the performance characteristics of the
previous implementation of parallel module loading. Testing on targets
with between 5000 and 14000 modules, I saw similar numbers as before,
often more than 10% faster in the new implementation across multiple
trials for these massive targets. I think it's because we have less lock
contention with this approach.
# The deadlock
See [bt.txt](https://github.com/user-attachments/files/22524471/bt.txt)
for a sample backtrace of LLDB when the deadlock occurs.
As @GeorgeHuyubo explains in his
[PR](https://github.com/llvm/llvm-project/pull/160225), the deadlock
occurs from an ABBA deadlock that happens when a thread context-switches
out of `Module::PreloadSymbols`, goes into `Target::GetOrCreateModule`
for another module, possibly entering this block:
```
if (!module_sp) {
// The platform is responsible for finding and caching an appropriate
// module in the shared module cache.
if (m_platform_sp) {
error = m_platform_sp->GetSharedModule(
module_spec, m_process_sp.get(), module_sp, &search_paths,
&old_modules, &did_create_module);
} else {
error = Status::FromErrorString("no platform is currently set");
}
}
```
`Module::PreloadSymbols` holds a module-level mutex, and then
`GetSharedModule` *attempts* to hold the mutex of the global shared
`ModuleList`. So, this thread holds the module mutex, and waits on the
global shared `ModuleList` mutex.
A competing thread may execute `Target::GetOrCreateModule`, enter the
same block as above, grabbing the global shared `ModuleList` mutex.
Then, in `ModuleList::GetSharedModule`, we eventually call
`ModuleList::FindModules` which eventually waits for the `Module` mutex
held by the first thread (via `Module::GetUUID`). Thus, we deadlock.
## Reproducing the deadlock
It might be worth noting that I've never been able to observe this
deadlock issue during live debugging (e.g. launching or attaching to
processes), however we were able to consistently reproduce this issue
with coredumps when using the following settings:
```
(lldb) settings set target.parallel-module-load true
(lldb) settings set target.preload-symbols true
(lldb) settings set symbols.load-on-demand false
(lldb) target create --core /some/core/file/here
# deadlock happens
```
## How this change avoids this deadlock
This change avoids concurrent executions of `Module::PreloadSymbols`
with `Target::GetOrCreateModule` by waiting until after the
`Target::GetOrCreateModule` executions to run `Module::PreloadSymbols`
in parallel. This avoids the ordering of holding a Module lock *then*
the ModuleList lock, as `Target::GetOrCreateModule` executions maintain
the ordering of the shared ModuleList lock first (from what I've read
and tested).
## Why not read-write lock?
Some feedback in https://github.com/llvm/llvm-project/pull/160225 was to
modify mutexes used in these components with read-write locks. This
might be a good idea overall, but I don't think it would *easily*
resolve this specific deadlock. `Module::PreloadSymbols` would probably
need a write lock to Module, so even if we had a read lock in
`Module::GetUUID` we would still contend. Maybe the `ModuleList` lock
could be a read lock that converts to a write lock if it chooses to
update the module, but it seems likely that some thread would try to
update the shared module list and then the write lock would contend
again.
Perhaps with deeper architectural changes, we could fix this issue?
# Other attempts
One downside of this approach (and the former approach of parallel
module loading) is that each DYLD would need to implement this pattern
themselves. With @clayborg's help, I looked at a few other approaches:
* In `Target::GetOrCreateModule`, backgrounding the
`Module::PreloadSymbols` call by adding it directly to the thread pool
via `Debugger::GetThreadPool().async()`. This required adding a lock to
`Module::SetLoadAddress` (probably should be one there already) since
`ObjectFileELF::SetLoadAddress` is not thread-safe (updates sections).
Unfortunately, during execution, this causes the preload symbols to run
synchronously with `Target::GetOrCreateModule`, preventing us from truly
parallelizing the execution.
* In `Module::PreloadSymbols`, backgrounding the `symtab` and `sym_file`
`PreloadSymbols` calls individually, but similar issues as the above.
* Passing a callback function like
https://github.com/swiftlang/llvm-project/pull/10746 instead of the
boolean I use in this change. It's functionally the same change IMO,
with some design tradeoffs:
* Pro: the caller doesn't need to explicitly call
`Module::PreloadSymbols` itself, and can instead call whatever function
is passed into the callback.
* Con: the caller needs to delay the execution of the callback such that
it occurs after the `GetOrCreateModule` logic, otherwise we run into the
same issue. I thought this would be trickier for the caller, requiring
some kinda condition variable or otherwise storing the calls to execute
afterwards.
# Test Plan:
```
ninja check-lldb
```
---------
Co-authored-by: Tom Yang <toyang@fb.com>
This patch extends ScriptedFrame to work with real (non-scripted)
threads,
enabling frame providers to synthesize frames for native processes.
Previously, ScriptedFrame only worked within
ScriptedProcess/ScriptedThread
contexts. This patch decouples ScriptedFrame from ScriptedThread,
allowing
users to augment or replace stack frames in real debugging sessions for
use
cases like custom calling conventions, reconstructing corrupted frames
from
core files, or adding diagnostic frames.
Key changes:
- ScriptedFrame::Create() now accepts ThreadSP instead of requiring
ScriptedThread, extracting architecture from the target triple rather
than ScriptedProcess.arch
- Added SBTarget::RegisterScriptedFrameProvider() and
ClearScriptedFrameProvider() APIs, with Target storing a
SyntheticFrameProviderDescriptor template for new threads
- Added "target frame-provider register/clear" commands for CLI access
- Thread class gains LoadScriptedFrameProvider(),
ClearScriptedFrameProvider(),
and GetFrameProvider() methods for per-thread frame provider management
- New SyntheticStackFrameList overrides FetchFramesUpTo() to lazily
provide
frames from either the frame provider or the real stack
This enables practical use of the SyntheticFrameProvider infrastructure
in
real debugging workflows.
rdar://161834688
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
We're iterating over the stop hooks so if one of them changes the stop
hook list by deleting itself or another stop hook, that invalidates our
iterator.
I chose to fix this by making a copy of the stop hook list and iterating
over that. That's a cheap operation since this is just an array of
shared pointers. But more importantly doing it this way means that if on
a stop where one stop hook deletes another, the deleted hook will always
get a chance to run. If you iterated over the list itself, then whether
that to be deleted hook gets to run would be dependent on whether it was
before or after the deleting stop hook, which would be confusing.
Main executables were bypassing the locate module callback that shared
libraries use, preventing custom symbol file location logic from working
consistently.
This PR fix this by
* Adding target context to ModuleSpec
* Leveraging that context to use target search path and platform's
locate module callback in ModuleList::GetSharedModule
This ensures both main executables and shared libraries get the same
callback treatment for symbol file resolution.
---------
Co-authored-by: George Hu <hyubo@meta.com>
Co-authored-by: George Hu <georgehuyubo@gmail.com>
This avoids unintentional comparisons between `SourceLanguage` and
`LanguageType`.
Also marks `operator bool` explicit so we don't implicitly convert to
bool.
This commit aggregates the following changes:
1. Fix the format (i.e., indentation) when printing stop hooks via `target
stop-hook list`.
2. Add `IndentScope Stream::MakeIndentScope()` to make managing (and restoring!)
of the indentation level on `Stream` instances more ergonomic and less error
prone.
3. Simplify printing of stop hooks using the new `IndentScope`.
### Summary
This PR refactors breakpoint event notification in LLDB to centralize
and eliminate code duplication. It creates a unified method in the
`Target` class for sending breakpoint change events. The new methods
check if listeners exist before broadcasting events
### Test
<img width="1532" height="76" alt="Screenshot 2025-10-23 at 12 49 31 PM"
src="https://github.com/user-attachments/assets/6d6a6da6-9684-463c-aeeb-90663cdbd077"
/>
---------
Co-authored-by: Piyush Jaiswal <piyushjais@meta.com>
Introduce the concept of internal stop hooks.
These are similar to LLDB's internal breakpoints:
LLDB itself will add them and users of LLDB will
not be able to add or remove them.
This change adds the following 3
independently-useful concepts:
* Maintain a list of internal stop hooks that will be populated by LLDB
and cannot be added to or removed from by users. They are managed in a
separate list in `Target::m_internal_stop_hooks`.
* `StopHookKind:CodeBased` and `StopHookCoded` represent a stop hook
defined by a C++ code callback (instead of command line expressions or a
Python class).
* Stop hooks that do not print any output can now also suppress the
printing of their header and description when they are hit via
`StopHook::GetSuppressOutput`.
Combining these 3 concepts we can model "internal
stop hooks" which serve the same function as
LLDB's internal breakpoints: executing built-in,
LLDB-defined behavior, leveraging the existing
mechanism of stop hooks.
This change also simplifies
`Target::RunStopHooks`. We already have to
materialize a new list for combining internal and
user stop hooks. Filter and only add active hooks to this list to avoid
the need for "isActive?"
checks later on.
This fixes a potential use after free where
ModuleList::RemoveSharedModuleIfOrphaned ->
SharedModuleList::RemoveIfOrphaned -> SharedModuleList::RemoveFromMap
would potentially dereference a freed pointer. This fixes it by not
calling ModuleList::RemoveSharedModuleIfOrphaned at all if the pointer
was just freed.
### Summary
Add support for unique target ids per Target instance. This is needed
for upcoming changes to allow debugger instances to be shared across
separate DAP instances for child process debugging. We want the IDE to
be able to attach to existing targets in an already runny lldb-dap
session, and having a unique ID per target would make that easier.
Each Target instance will have its own unique id, and uses a
function-local counter in `TargetList::CreateTargetInternal` to assign
incremental unique ids.
### Tests
Added several unit tests to test basic functionality, uniqueness of
targets, and target deletion doesn't affect the uniqueness.
```
bin/lldb-dotest -p TestDebuggerAPI
```
Resolves#141955
- Adds data to breakpoints `Source` object, in order for assembly
breakpoints, which rely on a temporary `sourceReference` value, to be
able to resolve in future sessions like normal path+line breakpoints
- Adds optional `instructions_offset` parameter to `BreakpointResolver`
Target::ReadMemory may or may not read live memory, but whether it did
read from live memory or from the filecache is opaque to callers. Add an
extra out parameter to indicate whether live memory was read or not.
c72c0b298c13 fixed a race condition in Target::GetExecutableModule. The
patch originally added the lock_guard but I suggested using the locking
ModuleList::Modules() helper instead. That didn't consider that the
fallback would still access the ModuleList without holding the lock.
This patch fixes the remaining issue.
…ess" (#144810)
This relands commit e0933ab5ae4856c4aa188a5ea16716b3a8d0840b. The
original commit was causing the test TestCreateAfterAttach.py to fail on
ARM Ubuntu bots. It's possible that this could've been happening because
the test for wait-attach progress reporting is waiting on a process
named "a.out" which could be too generic as multiple other tests (when
run in parallel on the bots) could also be using processes named
"a.out". This commit changes the wait-attach progress report test to
wait on a unique process name.
Original PR description:
This commit adds a progress report when wait-attaching to a process as
well as a test for this.
Original PR link: https://github.com/llvm/llvm-project/pull/144768
This is breaking TestCreateAfterAttach.py on Ubuntu:
```
======================================================================
FAIL: test_create_after_attach_dwo (TestCreateAfterAttach.CreateAfterAttachTestCase.test_create_after_attach_dwo)
Test thread creation after process attach.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/worker/as-builder-9/lldb-remote-linux-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1804, in test_method
return attrvalue(self)
^^^^^^^^^^^^^^^
File "/home/buildbot/worker/as-builder-9/lldb-remote-linux-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 149, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/worker/as-builder-9/lldb-remote-linux-ubuntu/llvm-project/lldb/test/API/functionalities/thread/create_after_attach/TestCreateAfterAttach.py", line 36, in test_create_after_attach
self.runCmd("process attach -p " + str(pid))
File "/home/buildbot/worker/as-builder-9/lldb-remote-linux-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1005, in runCmd
self.assertTrue(self.res.Succeeded(), msg + output)
AssertionError: False is not true : Command 'process attach -p 1474309' did not return successfully
Error output:
error: attach failed: lost connection
```
on the buildbots for lldb-remote-linux-ubuntu, lldb-arm-ubuntu,
lldb-aarch64-ubuntu, lldb-arm-ubuntu.
Since this PR: https://github.com/llvm/llvm-project/pull/141670/ We
started to override the Platform/Arch for a target if needed. However we
may have already registered locate module callback with the old
platform.
This PR will move the locate module callback to the new Platform
whenever Target changes architecture.
Co-authored-by: George Hu <georgehuyubo@gmail.com>
We're reading from the object's vtable to determine the pointer to the
full object. The vtable is normally in the "rodata" section of the
executable, which is often not included in the core file because it's
not supposed to change and the debugger can extrapolate its contents
from the executable file. We weren't doing that.
This patch changes the read operation to use the target class (which
falls back onto the executable module as expected) and adds the missing
ReadSignedIntegerFromMemory API. The fix is tested by creating a core
(minidump) file which deliberately omits the vtable pointer.
stop-hooks are supposed to trigger every time the process stops, but as
initially implemented they would only fire when control was returned to
the user. So for instance when a process was launched the stop hook
would only trigger when the process hit a breakpoint or crashed.
However, it would be really useful to be able to trigger a stop hook
when lldb first gains control over the process. One way to do that would
be to implement general "target lifecycle events" and then send process
created events that users could bind actions to.
OTOH, extending the stop hooks to fire when lldb first gains control
over the process is a pretty natural extension to the notion of a stop
hook. So this patch takes the shorter route to that ability by making
stop-hooks fire when lldb first gains control over the process.
I also added the ability to specify whether to trigger the stop hook "on
gaining control". I'm on the fence about whether to set the default to
be "trigger on gaining control" or "don't trigger on gaining control".
Since I think it's a generally useful feature, I've set the default to
"trigger on gaining control".
This patch improves LLDB launch time on Linux machines for **preload
scenarios**, particularly for executables with a lot of shared library
dependencies (or modules). Specifically:
* Launching a binary with `target.preload-symbols = true`
* Attaching to a process with `target.preload-symbols = true`.
It's completely controlled by a new flag added in the first commit
`plugin.dynamic-loader.posix-dyld.parallel-module-load`, which *defaults
to false*. This was inspired by similar work on Darwin #110646.
Some rough numbers to showcase perf improvement, run on a very beefy
machine:
* Executable with ~5600 modules: baseline 45s, improvement 15s
* Executable with ~3800 modules: baseline 25s, improvement 10s
* Executable with ~6650 modules: baseline 67s, improvement 20s
* Executable with ~12500 modules: baseline 185s, improvement 85s
* Executable with ~14700 modules: baseline 235s, improvement 120s
A lot of targets we deal with have a *ton* of modules, and unfortunately
we're unable to convince other folks to reduce the number of modules, so
performance improvements like this can be very impactful for user
experience.
This patch achieves the performance improvement by parallelizing
`DynamicLoaderPOSIXDYLD::RefreshModules` for the launch scenario, and
`DynamicLoaderPOSIXDYLD::LoadAllCurrentModules` for the attach scenario.
The commits have some context on their specific changes as well --
hopefully this helps the review.
# More context on implementation
We discovered the bottlenecks by via `perf record -g -p <lldb's pid>` on
a Linux machine. With an executable known to have 1000s of shared
library dependencies, I ran
```
(lldb) b main
(lldb) r
# taking a while
```
and showed the resulting perf trace (snippet shown)
```
Samples: 85K of event 'cycles:P', Event count (approx.): 54615855812
Children Self Command Shared Object Symbol
- 93.54% 0.00% intern-state libc.so.6 [.] clone3
clone3
start_thread
lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) r
std::_Function_handler<void* (), lldb_private::Process::StartPrivateStateThread(bool)::$_0>::_M_invoke(std::_Any_data const&)
lldb_private::Process::RunPrivateStateThread(bool) n
- lldb_private::Process::HandlePrivateEvent(std::shared_ptr<lldb_private::Event>&)
- 93.54% lldb_private::Process::ShouldBroadcastEvent(lldb_private::Event*)
- 93.54% lldb_private::ThreadList::ShouldStop(lldb_private::Event*)
- lldb_private::Thread::ShouldStop(lldb_private::Event*) *
- 93.53% lldb_private::StopInfoBreakpoint::ShouldStopSynchronous(lldb_private::Event*) t
- 93.52% lldb_private::BreakpointSite::ShouldStop(lldb_private::StoppointCallbackContext*) i
lldb_private::BreakpointLocationCollection::ShouldStop(lldb_private::StoppointCallbackContext*) k
lldb_private::BreakpointLocation::ShouldStop(lldb_private::StoppointCallbackContext*) b
lldb_private::BreakpointOptions::InvokeCallback(lldb_private::StoppointCallbackContext*, unsigned long, unsigned long) i
DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit(void*, lldb_private::StoppointCallbackContext*, unsigned long, unsigned lo
- DynamicLoaderPOSIXDYLD::RefreshModules() O
- 93.42% DynamicLoaderPOSIXDYLD::RefreshModules()::$_0::operator()(DYLDRendezvous::SOEntry const&) const u
- 93.40% DynamicLoaderPOSIXDYLD::LoadModuleAtAddress(lldb_private::FileSpec const&, unsigned long, unsigned long, bools
- lldb_private::DynamicLoader::LoadModuleAtAddress(lldb_private::FileSpec const&, unsigned long, unsigned long, boos
- 83.90% lldb_private::DynamicLoader::FindModuleViaTarget(lldb_private::FileSpec const&) o
- 83.01% lldb_private::Target::GetOrCreateModule(lldb_private::ModuleSpec const&, bool, lldb_private::Status*
- 77.89% lldb_private::Module::PreloadSymbols()
- 44.06% lldb_private::Symtab::PreloadSymbols()
- 43.66% lldb_private::Symtab::InitNameIndexes()
...
```
We saw that majority of time was spent in `RefreshModules`, with the
main culprit within it `LoadModuleAtAddress` which eventually calls
`PreloadSymbols`.
At first, `DynamicLoaderPOSIXDYLD::LoadModuleAtAddress` appears fairly
independent -- most of it deals with different files and then getting or
creating Modules from these files. The portions that aren't independent
seem to deal with ModuleLists, which appear concurrency safe. There were
members of `DynamicLoaderPOSIXDYLD` I had to synchronize though: namely
`m_loaded_modules` which `DynamicLoaderPOSIXDYLD` maintains to map its
loaded modules to their link addresses. Without synchronizing this, I
ran into SEGFAULTS and other issues when running `check-lldb`. I also
locked the assignment and comparison of `m_interpreter_module`, which
may be unnecessary.
# Alternate implementations
When creating this patch, another implementation I considered was
directly background-ing the call to `Module::PreloadSymbol` in
`Target::GetOrCreateModule`. It would have the added benefit of working
across platforms generically, and appeared to be concurrency safe. It
was done via `Debugger::GetThreadPool().async` directly. However, there
were a ton of concurrency issues, so I abandoned that approach for now.
# Testing
With the feature active, I tested via `ninja check-lldb` on both Debug
and Release builds several times (~5 or 6 altogether?), and didn't spot
additional failing or flaky tests.
I also tested manually on several different binaries, some with around
14000 modules, but just basic operations: launching, reaching main,
setting breakpoint, stepping, showing some backtraces.
I've also tested with the flag off just to make sure things behave
properly synchronously.
Introduce `StopHookResult::NoPreference` and
simplify control flow in `Target::RunStopHooks()`.
The algorithm is (in order):
1. "Auto continue" set on any hook -> continue
2. "Stop demanded" by any hook -> stop
3. "Continue requested" by any hook -> continue
4. No hooks, or "no preference" only (default
stance) -> stop
The new `NoPreference` lets us keep the default
stance, distinguishing case 3. and 4.
We didn't also copy over the next stop hook id, which meant we would
overwrite the stop hooks from the dummy target with stop hooks set after
they are copied over.
Remove Debugger::GetOutputStream and Debugger::GetErrorStream in
preparation for replacing both with a new variant that needs to be
locked and hence can't be handed out like we do right now.
The patch replaces most uses with GetAsyncOutputStream and
GetAsyncErrorStream respectively. There methods return new StreamSP
objects that automatically get flushed on destruction.
See #126630 for more details.
