In order to provide simple scripting support on top of instruction traces, a simple solution is to enhance the `dump instructions` command and allow printing in json and directly to a file. The format is verbose and not space efficient, but it's not supposed to be used for really large traces, in which case the TraceCursor API is the way to go.
- add a -j option for printing the dump in json
- add a -J option for pretty printing the json output
- add a -F option for specifying an output file
- add a -a option for dumping all the instructions available starting at the initial point configured with the other flags
- add tests for all cases
- refactored the instruction dumper and abstracted the actual "printing" logic. There are two writer implementations: CLI and JSON. This made the dumper itself much more readable and maintanable
sample output:
```
(lldb) thread trace dump instructions -t -a --id 100 -J
[
{
"id": 100,
"tsc": "43591204528448966"
"loadAddress": "0x407a91",
"module": "a.out",
"symbol": "void std::deque<Foo, std::allocator<Foo>>::_M_push_back_aux<Foo>(Foo&&)",
"mnemonic": "movq",
"source": "/usr/include/c++/8/bits/deque.tcc",
"line": 492,
"column": 30
},
...
```
Differential Revision: https://reviews.llvm.org/D128316
Add a test verifying that plain 'D' packet correctly detaches all
processes.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D127291
Refactor the fork and vfork tests to reuse the code better, avoid
unnecessary regexps and avoid unnecessary conversions between
hex-strings and integers.
Verify the server state after detaching. In particular, verify that
the detached process' PID/TID pair is no longer valid,
and that the correct process remains running.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D127290
Fix ThreadStopInfo struct to include the signal number for all events.
Since signo was not included in the details for fork, vfork
and vforkdone stops, the code incidentally referenced the wrong union
member, resulting in wrong signo being sent.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D127193
Implement the support for %Stop asynchronous notification packet format
in LLGS. This does not implement full support for non-stop mode for
threaded programs -- process plugins continue stopping all threads
on every event. However, it will be used to implement asynchronous
events in multiprocess debugging.
The non-stop protocol is enabled using QNonStop packet. When it is
enabled, the server uses notification protocol instead of regular stop
replies. Since all threads are always stopped, notifications are always
generated for all active threads and copied into stop notification
queue.
If the queue was empty, the initial asynchronous %Stop notification
is sent to the client immediately. The client needs to (eventually)
acknowledge the notification by sending the vStopped packet, in which
case it is popped from the queue and the stop reason for the next thread
is reported. This continues until notification queue is empty again,
in which case an OK reply is sent.
Asychronous notifications are also used for vAttach results and program
exits. The `?` packet uses a hybrid approach -- it returns the first
stop reason synchronously, and exposes the stop reasons for remaining
threads via vStopped queue.
The change includes a test case for a program generating a segfault
on 3 threads. The server is expected to generate a stop notification
for the segfaulting thread, along with the notifications for the other
running threads (with "no stop reason"). This verifies that the stop
reasons are correctly reported for all threads, and that notification
queue works.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D125575
...type variable by dereferencing the variable before
evaluating the expression.
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D128126
This patch implements VSCode DAP logpoints feature (also called tracepoint
in other VS debugger).
This will provide a convenient way for user to do printf style logging
debugging without pausing debuggee.
Differential Revision: https://reviews.llvm.org/D127702
Add trace load functionality to SBDebugger via the `LoadTraceFromFile` method.
Update intelpt test case class to have `testTraceLoad` method so we can take advantage of
the testApiAndSB decorator to test both the CLI and SB without duplicating code.
Differential Revision: https://reviews.llvm.org/D128107
Fix test_platform_file_fstat to correctly truncate/max out the expected
value when GDB Remote Serial Protocol specifies a value as an unsigned
integer but the underlying platform type uses a signed integer.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D128042
Make the AVX/MPX register tests more robust by checking for the presence
of actual registers rather than register sets. Account for the option
that the respective registers are defined but not available, as is
the case on FreeBSD and NetBSD. This fixes test regression on these
platforms.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D128041
Include the process identifier in the `T` stop responses when
multiprocess extension is enabled (i.e. prepend it to the thread
identifier). Use the exposed identifier to simplify the fork-and-follow
tests.
The LLDB client accounts for the possible PID since the multiprocess
extension support was added in b601c6719226fb83c43dae62a581e5ee08bfb169.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D127192
Include the process identifier in W/X stop reasons when multiprocess
extensions are enabled.
The LLDB client does not support process identifiers there at the moment
but it parses packets in such a way that their presence does not cause
any problems.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.llvm.org/D127191
Eliminate boilerplate of having each test manually assign to `mydir` by calling
`compute_mydir` in lldbtest.py.
Differential Revision: https://reviews.llvm.org/D128077
llvm's JSON parser supports 64 bit integers, but other tools like the
ones written in JS don't support numbers that big, so we need to
represent these possibly big numbers as a string. This diff uses that to
represent addresses and tsc zero. The former is printed in hex for and
the latter in decimal string form. The schema was updated mentioning
that.
Besides that, I fixed some remaining issues and now all test pass. Before I wasn't running all tests because for some reason my computer reverted perf_paranoid to 1.
Differential Revision: https://reviews.llvm.org/D127819
As discusses offline with @jj10305, we are updating some naming used throughout the code, specially in the json schema
- traceBuffer -> iptTrace
- core -> cpu
Differential Revision: https://reviews.llvm.org/D127817
This improves several things and addresses comments up to the diff [11] in this stack.
- Simplify many functions to receive less parameters that they can identify easily
- Create Storage classes for Trace and TraceIntelPT that can make it easier to reason about what can change with live process refreshes and what cannot.
- Don't cache the perf zero conversion numbers in lldb-server to make sure we get the most up-to-date numbers.
- Move the thread identifaction from context switches to the bundle parser, to leave TraceIntelPT simpler. This also makes sure that the constructor of TraceIntelPT is invoked when the entire data has been checked to be correct.
- Normalize all bundle paths before the Processes, Threads and Modules are created, so that they can assume that all paths are correct and absolute
- Fix some issues in the tests. Now they all pass.
- return the specific instance when constructing PerThread and MultiCore processor tracers.
- Properly implement IntelPTMultiCoreTrace::TraceStart.
- Improve some comments.
- Use the typedef ContextSwitchTrace more often for clarity.
- Move CreateContextSwitchTracePerfEvent to Perf.h as a utility function.
- Synchronize better the state of the context switch and the intel pt
perf events.
- Use a booblean instead of an enum for the PerfEvent state.
Differential Revision: https://reviews.llvm.org/D127456
For some context, The context switch data contains information of which threads were
executed by each traced process, therefore it's not necessary to specify
them in the trace file.
So this diffs adds support for that automatic feature. Eventually we
could include it to live processes as well.
Differential Revision: https://reviews.llvm.org/D127001
The process triple should only be needed when LLDB can't identify the correct
triple on its own. Examples could be universal mach-o binaries. But in any case,
at least for most of ELF files, LLDB should be able to do the job without having
the user specify the triple manually.
Differential Revision: https://reviews.llvm.org/D126990
This is the final functional patch to support intel pt decoding per cpu.
It works by doing the following:
- First, all context switches are split by tid and sorted in order. This produces a list of continuous executes per thread per core.
- Then, all intel pt subtraces are split by PSB boundaries and assigned to individual thread continuous executions on the same core by doing simple TSC-based comparisons.
- With this, we have, per thread, a sorted list of continuous executions each one with a list of intel pt subtraces. Up to this point, this is really fast because no instructions were actually decoded.
- Then, each thread can be decoded by traversing their continuous executions and intel pt subtraces. An advantage of having these continuous executions is that we can identify if a continuous exexecution doesn't have intel pt data, and thus has a gap in it. We can later to more sofisticated comparisons to identify if within a continuous execution there are gaps.
I'm adding a test as well.
Differential Revision: https://reviews.llvm.org/D126394
- Add the logic that parses all cpu context switch traces and produces blocks of continuous executions, which will be later used to assign intel pt subtraces to threads and to identify gaps. This logic can also identify if the context switch trace is malformed.
- The continuous executions blocks are able to indicate when there were some contention issues when producing the context switch trace. See the inline comments for more information.
- Update the 'dump info' command to show information and stats related to the multicore decoding flow, including timing about context switch decoding.
- Add the logic to conver nanoseconds to TSCs.
- Fix a bug when returning the context switches. Now they data returned makes sense and even empty traces can be returned from lldb-server.
- Finish the necessary bits for loading and saving a multi-core trace bundle from disk.
- Change some size_t to uint64_t for compatibility with 32 bit systems.
Tested by saving a trace session of a program that sleeps 100 times, it was able to produce the following 'dump info' text:
```
(lldb) trace load /tmp/trace3/trace.json (lldb) thread trace dump info Trace technology: intel-pt
thread #1: tid = 4192415
Total number of instructions: 1
Memory usage:
Total approximate memory usage (excluding raw trace): 2.51 KiB
Average memory usage per instruction (excluding raw trace): 2573.00 bytes
Timing for this thread:
Timing for global tasks:
Context switch trace decoding: 0.00s
Events:
Number of instructions with events: 0
Number of individual events: 0
Multi-core decoding:
Total number of continuous executions found: 2499
Number of continuous executions for this thread: 102
Errors:
Number of TSC decoding errors: 0
```
Differential Revision: https://reviews.llvm.org/D126267
:q!
This diff is massive, but it's because it connects the client with lldb-server
and also ensures that the postmortem case works.
- Flatten the postmortem trace schema. The reason is that the schema has become quite complex due to the new multicore case, which defeats the original purpose of having a schema that could work for every trace plug-in. At this point, it's better that each trace plug-in defines it's own full schema. This means that the only common field is "type".
-- Because of this new approach, I merged the "common" trace load and saving functionalities into the IntelPT one. This simplified the code quite a bit. If we eventually implement another trace plug-in, we can see then what we could reuse.
-- The new schema, which is flattened, has now better comments and is parsed better. A change I did was to disallow hex addresses, because they are a bit error prone. I'm asking now to print the address in decimal.
-- Renamed "intel" to "GenuineIntel" in the schema because that's what you see in /proc/cpuinfo.
- Implemented reading the context switch trace data buffer. I had to do
some refactors to do that cleanly.
-- A major change that I did here was to simplify the perf_event circular buffer reading logic. It was too complex. Maybe the original Intel author had something different in mind.
- Implemented all the necessary bits to read trace.json files with per-core data.
- Implemented all the necessary bits to save to disk per-core trace session.
- Added a test that ensures that parsing and saving to disk works.
Differential Revision: https://reviews.llvm.org/D126015
- Add a warnings field in the jLLDBGetState response, for warnings to be delivered to the client for troubleshooting. This removes the need to silently log lldb-server's llvm::Errors and not expose them easily to the user
- Simplify the tscPerfZeroConversion struct and schema. It used to extend a base abstract class, but I'm doubting that we'll ever add other conversion mechanisms because all modern kernels support perf zero. It is also the one who is supposed to work with the timestamps produced by the context switch trace, so expecting it is imperative.
- Force tsc collection for cpu tracing.
- Add a test checking that tscPerfZeroConversion is returned by the GetState request
- Add a pre-check for cpu tracing that makes sure that perf zero values are available.
Differential Revision: https://reviews.llvm.org/D125932
- Add collection of context switches per cpu grouped with the per-cpu intel pt traces.
- Move the state handling from the interl pt trace class to the PerfEvent one.
- Add support for stopping and enabling perf event groups.
- Return context switch entries as part of the jLLDBTraceGetState response.
- Move the triggers of whenever the process stopped or resumed. Now the will-resume notification is in a better location, which will ensure that we'll capture the instructions that will be executed.
- Remove IntelPTSingleBufferTraceUP. The unique pointer was useless.
- Add unit tests
Differential Revision: https://reviews.llvm.org/D125897
We have two different "process trace" implementations: per thread and per core. As a way to simplify the collector who uses both, I'm creating a base abstract class that is used by these implementations. This effectively simplify a good chunk of code.
Differential Revision: https://reviews.llvm.org/D125503
NativeProcessLinux is not able to properly read libraries-svr4 data when
running with ld.so as the "main" executable. Normally, this is not a big
problem, as it returns an error message, and lldb can fallback to manual
library loading.
Unfortunately, lldb-server also does not clear cached svr4 data on exec,
which means that it does *not* return an error when the application
execs from the "regular" to the "ld.so" mode. Instead it returns
incorrect data (it is missing the main executable) and causes
TestDyldExecLinux to fail (but only when building with xml support
enabled).
This patch makes ensures that cached process data is cleared on exec,
fixing the test. Since TestDyldExecLinux has shown to be sensitive to
the way we read library info, I fork it into two (with svr4 enabled and
disabled).
This patch adds a libcxx formatter for std::span. The
implementation is based on the libcxx formatter for
std::vector. The main difference is the fact that
std::span conditionally has a __size member based
on whether it has a static or dynamic extent.
Example output of formatted span:
(std::span<const int, 18446744073709551615>) $0 = size=6 {
[0] = 0
[1] = 1
[2] = 2
[3] = 3
[4] = 4
[5] = 5
}
The second template parameter here is actually std::dynamic_extent,
but the type declaration we get back from the TypeSystemClang is the
actual value (which in this case is (size_t)-1). This is consistent
with diagnostics from clang, which doesn't desugar this value either.
E.g.,:
span.cpp:30:31: error: implicit instantiation of undefined template
'Undefined<std::span<int, 18446744073709551615>>'
Testing:
Added API-tests
Confirmed manually using LLDB cli that printing spans works in various scenarios
Patch by Michael Buch!
Differential Revision: https://reviews.llvm.org/D127481
When using the `script` Python repl, SB objects are printed in a way that gives
the user no information. The simplest example is:
```
(lldb) script lldb.debugger
<lldb.SBDebugger; proxy of <Swig Object of type 'lldb::SBDebugger *' at 0x1097a5de0> >
```
This output comes from the Python repl printing the `repr()` of an object.
None of the SB classes implement `__repr__`, and all print like the above.
However, many (most?, all?) SB classes implement `__str__`. Because they
implement `__str__`, a more detailed output can be had by `print`ing the
object, for example:
```
(lldb) script print(lldb.debugger)
Debugger (instance: "debugger_1", id: 1)
```
For convenience, this change switches all SB classes that implement to
`__str__` to instead implement `__repr__`. **The result is that `str()` and
`repr()` will produce the same output**. This is because `str` calls `__repr__`
for classes that have no `__str__` method.
The benefit being that when writing a `script` invocation, you don't need to
remember to wrap in `print()`. If that isn't enough motivation, consider the
case where your Python expression results in a list of SB objects, in that case
you'd have to `map` or use a list comprehension like `[str(x) for x in <expr>]`
in order to see the details of the objects in the list.
For reference, the docs for `repr` say:
> repr(object)
> Return a string containing a printable representation of an object. For
> many types, this function makes an attempt to return a string that would
> yield an object with the same value when passed to eval(); otherwise, the
> representation is a string enclosed in angle brackets that contains the
> name of the type of the object together with additional information often
> including the name and address of the object. A class can control what this
> function returns for its instances by defining a __repr__() method.
and the docs for `__repr__` say:
> object.__repr__(self)
> Called by the repr() built-in function to compute the “official” string
> representation of an object. If at all possible, this should look like a
> valid Python expression that could be used to recreate an object with the
> same value (given an appropriate environment). If this is not possible, a
> string of the form <...some useful description...> should be returned. The
> return value must be a string object. If a class defines __repr__() but not
> __str__(), then __repr__() is also used when an “informal” string
> representation of instances of that class is required.
>
> This is typically used for debugging, so it is important that the
> representation is information-rich and unambiguous.
Even if it were convenient to construct Python expressions for SB classes so
that they could be `eval`'d, however for typical lldb usage, I can't think of a
motivating reason to do so. As it stands, the only action the docs say to do,
that this change doesn't do, is wrap the `repr` string in `<>` angle brackets.
An alternative implementation is to change lldb's python repl to apply `str()`
to the top level result. While this would work well in the case of a single SB
object, it doesn't work for a list of SB objects, since `str([x])` uses `repr`
to convert each list element to a string.
Differential Revision: https://reviews.llvm.org/D127458
Add a function to make it easier to debug a test failure caused by an
unexpected state.
Currently, tests are using assertEqual which results in a cryptic error
message: "AssertionError: 5 != 10". Even when a test provides a message
to make it clear why a particular state is expected, you still have to
figure out which of the two was the expected state, and what the other
value corresponds to.
We have a function in lldbutil that helps you convert the state number
into a user readable string. This patch adds a wrapper around
assertEqual specifically for comparing states and reporting better error
messages.
The aforementioned error message now looks like this: "AssertionError:
stopped (5) != exited (10)". If the user provided a message, that
continues to get printed as well.
Differential revision: https://reviews.llvm.org/D127355
On macOS Ventura and later, dyld and the main binary will be loaded
again when dyld moves itself into the shared cache. Update the test
accordingly.
Differential revision: https://reviews.llvm.org/D127331
In order to avoid stranding the Objective-C runtime lock, we switched
from objc_copyRealizedClassList to its non locking variant
objc_copyRealizedClassList_nolock. Not taking the lock was relatively
safe because we run this expression on one thread only, but it was still
possible that someone was in the middle of modifying this list while we
were trying to read it. Worst case that would result in a crash in the
inferior without side-effects and we'd unwind and try again later.
With the introduction of macOS Ventura, we can use
objc_getRealizedClassList_trylock instead. It has semantics similar to
objc_copyRealizedClassList_nolock, but instead of not locking at all,
the function returns if the lock is already taken, which avoids the
aforementioned crash without stranding the Objective-C runtime lock.
Because LLDB gets to allocate the underlying memory we also avoid
stranding the malloc lock.
rdar://89373233
Differential revision: https://reviews.llvm.org/D127252
A non-zero exit code from the test binary results in a
CalledProcessError. Without catching the exception, that would result in
a error (unresolved test) instead of a failure. This patch fixes that.
Currently we look for keywords in the dotest.py output to determine the
lit result code. This binary approach of a keyword being present works
for PASS and FAIL, where having at least one test pass or fail
respectively results in that exit code. Things are more complicated
for tests that neither passed or failed, but report a combination of
(un)expected failures, skips or unresolved tests.
This patch changes the logic to parse the number of tests with a
particular result from the dotest.py output. For tests that did not PASS
or FAIL, we now report the lit result code for the one that occurred the
most. For example, if we had a test with 3 skips and 4 expected
failures, we report the test as XFAIL.
We're still mapping multiple tests to one result code, so some loss of
information is inevitable.
Differential revision: https://reviews.llvm.org/D127258
This test depends on multiple threads with one of them
hitting a watchpoint at the same time as a breakpoint, and
can fail because of the way arm64 watchpoints are handled.
I added skips to most of these via
```
commit bef4da4a6aef8196f007f44e3e9c8e3419ffb623
Author: Jason Molenda <jason@molenda.com>
Date: Wed May 25 16:05:16 2022 -0700
Skip testing of watchpoint hit-count/ignore-count on multithreaded
```
but missed that this test is susceptable to the same issue.
This patch remove XFAIL decorator from tests which as passing on AArch64
Windows. This is tested on surface pro x using tot llvm and clang 14.0.3
as compiler with visual studio 2019 x86_arm64 environment.
It was putting the command the user typed, and then the resolved command in the
command history. That caused up-arrow not to work correctly when the regex command
was invoked from a Python-command. Plus it's just weird.
Differential Revision: https://reviews.llvm.org/D126789
The function that was supposed to iterate over all the breakpoints sharing
BKPT_NAME stopped after the first one because of a reversed "if success"
condition.
Differential Revision: https://reviews.llvm.org/D126730
so that they can be used to prime new Process runs. "process handle"
was also changed to populate the dummy target if there's no selected
target, so that the settings will get copied into new targets.
Differential Revision: https://reviews.llvm.org/D126259
After changing the "fallback" behavior when a user sets a breakpoint
without specifying a module the bad-address-breakpoint test case failed
incorrectly. This patch updates that test case in order to more
thoroughly discover an illegal address and use that as the means for
testing whether a breakpoint set at an illegal address fails to resolve.
Differential revision: https://reviews.llvm.org/D126109
I get to my work directory through a symlink, so the pathnames the
tests get for their build artifacts etc are via that symlink. There
are three tests which compare those symlink paths to a directory
received from dyld on macOS, which is the actual real pathname.
These tests have always failed for me on my dekstop but I finally
sat down to figure out why. Easy quick fix.
