### Context
Over a year ago, I landed support for 64b Memory ranges in Minidump
(#95312). In this patch we added the Memory64 list stream, which is
effectively a Linked List on disk. The layout is a sixteen byte header
and then however many Memory descriptors.
### The Bug
This is a classic off-by one error, where I added 8 bytes instead of 16
for the header. This caused the first region to start 8 bytes before the
correct RVA, thus shifting all memory reads by 8 bytes. We are correctly
writing all the regions to disk correctly, with no physical corruption
but the RVA is defined wrong, meaning we were incorrectly reading memory
### Why wasn't this caught?
One problem we've had is forcing Minidump to actually use the 64b mode,
it would be a massive waste of resources to have a test that actually
wrote >4.2gb of IO to validate the 64b regions, and so almost all
validation has been manual. As a weakness of manual testing, this issue
is psuedo non-deterministic, as what regions end up in 64b or 32b is
handled greedily and iterated in the order it's laid out in
/proc/pid/maps. We often validated 64b was written correctly by
hexdumping the Minidump itself, which was not corrupted (other than the
BaseRVA)
### Why is this showing up now?
During internal usage, we had a bug report that the Minidump wasn't
displaying values. I was unable to repro the issue, but during my
investigation I saw the variables were in the 64b regions which resulted
in me identifying the bug.
### How do we prevent future regressions?
To prevent regressions, and honestly to save my sanity for figuring out
where 8 bytes magically came from, I've added a new API to
SBSaveCoreOptions.
```SBSaveCoreOptions::GetMemoryRegionsToSave()```
The ability to get the memory regions that we intend to include in the Coredump. I added this so we can compare what we intended to include versus what was actually included. Traditionally we've always had issues comparing regions because Minidump includes `/proc/pid/maps` and it can be difficult to know what memoryregion read failure was a genuine error or just a page that wasn't meant to be included.
We are also leveraging this API to choose the memory regions to be generated, as well as for testing what regions should be bytewise 1:1.
After much debate with @clayborg, I've moved all non-stack memory to the Memory64 List. This list doesn't incur us any meaningful overhead and Greg originally suggested doing this in the original 64b PR. This also means we're exercising the 64b path every single time we save a Minidump, preventing regressions on this feature from slipping through testing in the future.
Snippet produced by [minidump.py](https://github.com/clayborg/scripts)
```
MINIDUMP_MEMORY_LIST:
NumberOfMemoryRanges = 0x00000002
MemoryRanges[0] = [0x00007f61085ff9f0 - 0x00007f6108601000) @ 0x0003f655
MemoryRanges[1] = [0x00007ffe47e50910 - 0x00007ffe47e52000) @ 0x00040c65
MINIDUMP_MEMORY64_LIST:
NumberOfMemoryRanges = 0x000000000000002e
BaseRva = 0x0000000000042669
MemoryRanges[0] = [0x00005584162d8000 - 0x00005584162d9000)
MemoryRanges[1] = [0x00005584162d9000 - 0x00005584162db000)
MemoryRanges[2] = [0x00005584162db000 - 0x00005584162dd000)
MemoryRanges[3] = [0x00005584162dd000 - 0x00005584162ff000)
MemoryRanges[4] = [0x00007f6100000000 - 0x00007f6100021000)
MemoryRanges[5] = [0x00007f6108800000 - 0x00007f6108828000)
MemoryRanges[6] = [0x00007f6108828000 - 0x00007f610899d000)
MemoryRanges[7] = [0x00007f610899d000 - 0x00007f61089f9000)
MemoryRanges[8] = [0x00007f61089f9000 - 0x00007f6108a08000)
MemoryRanges[9] = [0x00007f6108bf5000 - 0x00007f6108bf7000)
```
### Misc
As a part of this fix I had to look at LLDB logs a lot, you'll notice I added `0x` to many of the PRIx64 `LLDB_LOGF`. This is so the user (or I) can directly copy paste the address in the logs instead of adding the hex prefix themselves.
Added some SBSaveCore tests for the new GetMemoryAPI, and Docstrings.
CC: @DavidSpickett, @da-viper @labath because we've been working together on save-core plugins, review it optional and I didn't tag you but figured you'd want to know
This commit adds completion support for the plugin commands. It will try
to complete partial namespaces to the full namespace string. If the
completion input is already a full namespace string then it will add all
the matching plugins in that namespace as completions.
This lets the user complete to the namespace first and then tab-complete
to the next level if desired.
```
(lldb) plugin list a<tab>
Available completions:
abi
architecture
(lldb) plugin list ab<tab>
(lldb) plugin list abi<tab>
(lldb) plugin list abi.<tab>
Available completions:
abi.SysV-arm64
abi.ABIMacOSX_arm64
abi.SysV-arm
...
```
This PR adds an MCP (Model Context Protocol ) server to LLDB. For
motivation and background, please refer to the corresponding RFC:
https://discourse.llvm.org/t/rfc-adding-mcp-support-to-lldb/86798
I implemented this as a new kind of plugin. The idea is that we could
support multiple protocol servers (e.g. if we want to support DAP from
within LLDB). This also introduces a corresponding top-level command
(`protocol-server`) with two subcommands to `start` and `stop` the
server.
```
(lldb) protocol-server start MCP tcp://localhost:1234
MCP server started with connection listeners: connection://[::1]:1234, connection://[127.0.0.1]:1234
```
The MCP sever supports one tool (`lldb_command`) which executes a
command, but can easily be extended with more commands.
In #134418 we added support to list/enable/disable `SystemRuntime` and
`InstrumentationRuntime` plugins. We limited it to those two plugin
types to flesh out the idea with a smaller change.
This PR adds support for the remaining plugin types. We now support all
the plugins that can be registered directly with the plugin manager.
Plugins that are added by loading shared objects are still not
supported.
This commit adds three new commands for managing plugins. The `list`
command will show which plugins are currently registered and their
enabled state. The `enable` and `disable` commands can be used to enable
or disable plugins.
A disabled plugin will not show up to the PluginManager when it iterates
over available plugins of a particular type.
The purpose of these commands is to provide more visibility into
registered plugins and allow users to disable plugins for experimental
perf reasons.
There are a few limitations to the current implementation
1. Only SystemRuntime and InstrumentationRuntime plugins are currently
supported. We can easily extend the existing implementation to support
more types. The scope was limited to these plugins to keep the PR size
manageable.
2. Only "statically" know plugin types are supported (i.e. those managed
by the PluginManager and not from `plugin load`). It is possibly we
could support dynamic plugins as well, but I have not looked into it
yet.
Identical PR to: https://github.com/llvm/llvm-project/pull/134563
Previous PR was approved and landed but broke the build due to bad
merge.
Manually resolve the merge conflict and try to land again.
Co-authored-by: George Hu <georgehuyubo@gmail.com>
This reverts commit 070a4ae2f9bcf6967a7147ed2972f409eaa7d3a6.
Multiple buildbot failures have been reported:
https://github.com/llvm/llvm-project/pull/134563
The build fails with:
lldb/source/Target/Statistics.cpp:75:39: error: use of undeclared
identifier 'num_symbols_loaded'
Adds the new `plugin.cplusplus.display.function-name-format` setting and makes the `${function.name-with-args}` query it for formatting the function name.
One caveat is that the setting can't itself be set to `${function.name-with-args}` because that would cause infinite recursion and blow the stack. I added an XFAILed test-case for it and will address it in a follow-up patch.
https://github.com/llvm/llvm-project/pull/131836
This commit adds support for enabling and disabling plugins by name. The
changes are made generically in the `PluginInstances` class, but
currently we only expose the ability to SystemRuntime plugins. Other
plugins types can be added easily.
We had a few design goals for how disabled plugins should work
1. Plugins that are disabled should still be visible to the system. This
allows us to dynamically enable and disable plugins and report their
state to the user.
2. Plugin order should be stable across disable and enable changes. We
want avoid changing the order of plugin lookup. When a plugin is
re-enabled it should return to its original slot in the creation order.
3. Disabled plugins should not appear in PluginManager operations.
Clients should be able to assume that only enabled plugins will be
returned from the PluginManager.
For the implementation we modify the plugin instance to maintain a bool
of its enabled state. Existing clients external to the Instances class
expect to iterate over only enabled instance so we skip over disabed
instances in the query and snapshot apis. This way the client does not
have to manually check which instances are enabled.
Remove raw access to PluginInstances vector
This commit modifies the PluginInstances class to remove direct access
to the m_instances vector. Instead, we expose a new `GetSnapshot` method
that returns a copy of the current state of the instances vector. All
external iteration over the instances is updated to use the new method.
The motivation for the change is to allow modifying the way we store
instances without having to change all the clients. This is a
preliminary change to allow enabling/disabling of plugins in which case
we want to iterate over only enabled plugins.
We also considered using a custom iterator that wraps the vector
iterator and can skip over disabled instances. That works, but the
iterator code is a bit messy with all template and typedefs to make a
compliant iterator.
This patch removes all of the Set.* methods from Status.
This cleanup is part of a series of patches that make it harder use the
anti-pattern of keeping a long-lives Status object around and updating
it while dropping any errors it contains on the floor.
This patch is largely NFC, the more interesting next steps this enables
is to:
1. remove Status.Clear()
2. assert that Status::operator=() never overwrites an error
3. remove Status::operator=()
Note that step (2) will bring 90% of the benefits for users, and step
(3) will dramatically clean up the error handling code in various
places. In the end my goal is to convert all APIs that are of the form
` ResultTy DoFoo(Status& error)
`
to
` llvm::Expected<ResultTy> DoFoo()
`
How to read this patch?
The interesting changes are in Status.h and Status.cpp, all other
changes are mostly
` perl -pi -e 's/\.SetErrorString/ = Status::FromErrorString/g' $(git
grep -l SetErrorString lldb/source)
`
plus the occasional manual cleanup.
Reapply #100443 and #101770. These were originally reverted due to a
test failure and an MSAN failure. I changed the test attribute to
restrict to x86 (following the other existing tests). I could not
reproduce the test or the MSAN failure and no repo steps were provided.
In #100443, Mach-o and Minidump now only call process API's that take a
`SaveCoreOption` as the container for the style and information if a
thread should be included in the core or not. This introduced a bug
where in subsequent method calls we were not honoring the defaults of
both implementations.
~~To solve this I have made a copy of each SaveCoreOptions that is
mutable by the respective plugin. Originally I wanted to leave the
SaveCoreOptions as non const so these default value mutations could be
shown back to the user. Changing that behavior is outside of the scope
of this bugfix, but is context for why we are making a copy.~~
Removed const on the savecoreoptions so defaults can be inspected by the
user
CC: @Michael137
In #98403 I enabled the SBSaveCoreOptions object, which allows users via
the scripting API to define what they want saved into their core file.
As the first option I've added a threadlist, so users can scan and
identify which threads and corresponding stacks they want to save.
In order to support this, I had to add a new method to `Process.h` on
how we identify which threads are to be saved, and I had to change the
book keeping in minidump to ensure we don't double save the stacks.
Important to @jasonmolenda I also changed the MachO coredump to accept
these new APIs.
This patch introduces a new `template` multiword sub-command to the
`scripting` top-level command. As the name suggests, this sub-command
operates on scripting templates, and currently has the ability to
automatically discover the various scripting extensions that lldb
supports.
This was previously reviewed in #97273.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch introduces a new `template` multiword sub-command to the
`scripting` top-level command. As the name suggests, this sub-command
operates on scripting templates, and currently has the ability to
automatically discover the various scripting extensions that lldb
supports.
This was previously reviewed in #97273.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch introduces a new `template` multiword sub-command to the
`scripting` top-level command. As the name suggests, this sub-command
operates on scripting templates, and currently has the ability to
automatically discover the various scripting extensions that lldb
supports.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This PR adds `SBSaveCoreOptions`, which is a container class for options
when LLDB is taking coredumps. For this first iteration this container
just keeps parity with the extant API of `file, style, plugin`. In the
future this options object can be extended to allow users to take a
subset of their core dumps.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
I've plumbed the LLVM DebugInfoD client into LLDB, and added automatic
downloading of DWP files to the SymbolFileDWARF.cpp plugin. If you have
DEBUGINFOD_URLS set to a space delimited set of web servers, LLDB will
try to use them as a last resort when searching for DWP files. If you do
*not* have that environment variable set, nothing should be changed.
There's also a setting, per @clayborg 's suggestion, that will override
the environment variable, or can be used instead of the environment
variable. The setting is why I also needed to add an API to the
llvm-debuginfod library
### Test Plan:
Suggestions are welcome here. I should probably have some positive and
negative tests, but I wanted to get the diff up for people who have a
clue what they're doing to rip it to pieces before spending too much
time validating the initial implementation.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
Co-authored-by: Alex Langford <nirvashtzero@gmail.com>
This builds on top of the work started in c3a302d to convert
LocateSymbolFile to a SymbolLocator plugin. This commit moves
DownloadObjectAndSymbolFile.
This commit contains the initial scaffolding to convert the
functionality currently implemented in LocateSymbolFile to a plugin
architecture. The plugin approach allows us to easily add new ways to
find symbols and fixes some issues with the current implementation.
For instance, currently we (ab)use the host OS to include support for
querying the DebugSymbols framework on macOS. The plugin approach
retains all the benefits (including the ability to compile this out on
other platforms) while maintaining a higher level of separation with the
platform independent code.
To limit the scope of this patch, I've only converted a single function:
LocateExecutableObjectFile. Future commits will convert the remaining
LocateSymbolFile functions and eventually remove LocateSymbolFile. To
make reviewing easier, that will done as follow-ups.
The underlying structures no longer use ConstString so we can remove it
wholesale from PluginManager now.
Differential Revision: https://reviews.llvm.org/D153818
I removed ConstString from OptionValueProperties in 643ba926c1f6, but
there are a few call sites that still create a ConstString as an
argument. I did not catch these initially because ConstString has an
implicit conversion method to StringRef.
Differential Revision: https://reviews.llvm.org/D153673
These probably do not need to be in the ConstString StringPool as they
don't really need any of the advantages that ConstStrings offer.
Lifetime for these things is always static and we never need to perform
comparisons for setting descriptions.
Differential Revision: https://reviews.llvm.org/D148679
This change uses the information from target.xml sent by
the GDB stub to produce C types that we can use to print
register fields.
lldb-server *does not* produce this information yet. This will
only work with GDB stubs that do. gdbserver or qemu
are 2 I know of. Testing is added that uses a mocked lldb-server.
```
(lldb) register read cpsr x0 fpcr fpsr x1
cpsr = 0x60001000
= (N = 0, Z = 1, C = 1, V = 0, TCO = 0, DIT = 0, UAO = 0, PAN = 0, SS = 0, IL = 0, SSBS = 1, BTYPE = 0, D = 0, A = 0, I = 0, F = 0, nRW = 0, EL = 0, SP = 0)
```
Only "register read" will display fields, and only when
we are not printing a register block.
For example, cpsr is a 32 bit register. Using the target's scratch type
system we construct a type:
```
struct __attribute__((__packed__)) cpsr {
uint32_t N : 1;
uint32_t Z : 1;
...
uint32_t EL : 2;
uint32_t SP : 1;
};
```
If this register had unallocated bits in it, those would
have been filled in by RegisterFlags as anonymous fields.
A new option "SetChildPrintingDecider" is added so we
can disable printing those.
Important things about this type:
* It is packed so that sizeof(struct cpsr) == sizeof(the real register).
(this will hold for all flags types we create)
* Each field has the same storage type, which is the same as the type
of the raw register value. This prevents fields being spilt over
into more storage units, as is allowed by most ABIs.
* Each bitfield size matches that of its register field.
* The most significant field is first.
The last point is required because the most significant bit (MSB)
being on the left/top of a print out matches what you'd expect to
see in an architecture manual. In addition, having lldb print a
different field order on big/little endian hosts is not acceptable.
As a consequence, if the target is little endian we have to
reverse the order of the fields in the value. The value of each field
remains the same. For example 0b01 doesn't become 0b10, it just shifts
up or down.
This is needed because clang's type system assumes that for a struct
like the one above, the least significant bit (LSB) will be first
for a little endian target. We need the MSB to be first.
Finally, if lldb's host is a different endian to the target we have
to byte swap the host endian value to match the endian of the target's
typesystem.
| Host Endian | Target Endian | Field Order Swap | Byte Order Swap |
|-------------|---------------|------------------|-----------------|
| Little | Little | Yes | No |
| Big | Little | Yes | Yes |
| Little | Big | No | Yes |
| Big | Big | No | No |
Testing was done as follows:
* Little -> Little
* LE AArch64 native debug.
* Big -> Little
* s390x lldb running under QEMU, connected to LE AArch64 target.
* Little -> Big
* LE AArch64 lldb connected to QEMU's GDB stub, which is running
an s390x program.
* Big -> Big
* s390x lldb running under QEMU, connected to another QEMU's GDB
stub, which is running an s390x program.
As we are not allowed to link core code to plugins directly,
I have added a new plugin RegisterTypeBuilder. There is one implementation
of this, RegisterTypeBuilderClang, which uses TypeSystemClang to build
the CompilerType from the register fields.
Reviewed By: jasonmolenda
Differential Revision: https://reviews.llvm.org/D145580
The low-level decoder might fall into an infinite decoding loop for
various reasons, the simplest being an infinite direct loop reached due
to wrong handling of self-modified code in the kernel, e.g. it might
reach
```
0x0A: pause
0x0C: jump to 0x0A
```
In this case, all the code is sequential and requires no packets to be
decoded. The low-level decoder would produce an output like the
following
```
0x0A: pause
0x0C: jump to 0x0A
0x0A: pause
0x0C: jump to 0x0A
0x0A: pause
0x0C: jump to 0x0A
... infinite amount of times
```
These cases require stopping the decoder to avoid infinite work and signal this
at least as a trace error.
- Add a check that breaks decoding of a single PSB once 500k instructions have been decoded since the last packet was processed.
- Add a check that looks for infinite loops after certain amount of instructions have been decoded since the last packet was processed.
- Add some `settings` properties for tweaking the thresholds of the checks above. This is also nice because it does the basic work needed for future settings.
- Add an AnomalyDetector class that inspects the DecodedThread and the libipt decoder in search for anomalies. These anomalies are then signaled as fatal errors in the trace.
- Add an ErrorStats class that keeps track of all the errors in a DecodedThread, with a special counter for fatal errors.
- Add an entry for decoded thread errors in the `dump info` command.
Some notes are added in the code and in the documention of the settings,
so please read them.
Besides that, I haven't been unable to create a test case in LLVM style, but
I've found an anomaly in the thread #12 of the trace
72533820-3eb8-4465-b8e4-4e6bf0ccca99 at Meta. We have to figure out how to
artificially create traces with this kind of anomalies in LLVM style.
With this change, that anomalous thread now shows:
```
(lldb)thread trace dump instructions 12 -e -i 23101
thread #12: tid = 8
...missing instructions
23101: (error) anomalous trace: possible infinite loop detected of size 2
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
23100: 0xffffffff81342785 pause
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
23099: 0xffffffff81342787 jmp 0xffffffff81342785 ; <+5> [inlined] rep_nop at processor.h:13:2
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
23098: 0xffffffff81342785 pause
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
23097: 0xffffffff81342787 jmp 0xffffffff81342785 ; <+5> [inlined] rep_nop at processor.h:13:2
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
23096: 0xffffffff81342785 pause
vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
23095: 0xffffffff81342787 jmp 0xffffffff81342785 ; <+5> [inlined] rep_nop at processor.h:13:2
```
It used to be in an infinite loop where the decoder never stopped.
Besides that, the dump info command shows
```
(lldb) thread trace dump info 12
Errors:
Number of individual errors: 32
Number of fatal errors: 1
Number of other errors: 31
```
and in json format
```
(lldb) thread trace dump info 12 -j
"errors": {
"totalCount": 32,
"libiptErrors": {},
"fatalErrors": 1,
"otherErrors": 31
}
```
Differential Revision: https://reviews.llvm.org/D136557
This patch teaches LLDB about Mach-O filesets. Filsets are Mach-O files
that contain a bunch of other Mach-O files. Unlike universal binaries,
which have a different header, Filesets use load commands to describe
the different entries it contains.
Differential revision: https://reviews.llvm.org/D132433
As previously discussed with @jj10306, we didn't really have a name for
the post-mortem (or offline) trace session representation, which is in
fact a folder with a bunch of files. We decided to call this folder
"trace bundle", and the main JSON file in it "trace bundle description
file". This naming is pretty decent, so I'm refactoring all the existing
code to account for that.
Differential Revision: https://reviews.llvm.org/D128484
PE/COFF can use either MSVC or GNU (MinGW) ABI for C++ code, however
LLDB had defaulted to MSVC implicitly with no way to override it. This
causes issues when debugging modules built with the GNU ABI, sometimes
even crashes.
This changes the PE/COFF plugin to set the module triple according to
the default target triple used to build LLDB. If the default target
triple is Windows and a valid environment is specified, then this
environment will be used for the module spec. This not only works for
MSVC and GNU, but also other environments.
A new setting, `plugin.object-file.pe-coff.abi`, has been added to
allow overriding this default ABI.
* Fixes https://github.com/llvm/llvm-project/issues/50775
* Fixes https://github.com/mstorsjo/llvm-mingw/issues/226
* Fixes https://github.com/mstorsjo/llvm-mingw/issues/282
Reviewed By: omjavaid
Differential Revision: https://reviews.llvm.org/D127048
Currently, we'll try to instantiate a ClangREPL for every known
language. The plugin manager already knows what languages it supports,
so rely on that to only instantiate a REPL when we know the requested
language is supported.
rdar://86439474
Differential revision: https://reviews.llvm.org/D115698
