shylie/llvm-project
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[lldb][Docs] Additions to debuging LLDB page (#65635)

Browse Source 
Adds the following:
* A note that you can use attaching to debug the right lldb-server
process, though there are drawbacks.
* A section on debugging the remote protocol.
* Reducing bugs, including reducing ptrace bugs to remove the need for
LLDB.

I've added a standlone ptrace program to the examples folder because:
* There's no better place to put it.
* Adding it to the page seems like wasting space, and would be harder to
update.
* I link to Eli Bendersky's classic blog on the subject, but we are
safer with our own example as well.
* Eli's example is for 32 bit Intel, AArch64 is more common these days.
* It's easier to show the software breakpoint steps in code than explain
it (though I still do that in the text).
* It was living on my laptop not helping anyone so I think it's good to
have it upstream for others, including future me.
This commit is contained in:
David Spickett 2023-09-08 10:05:16 +01:00 committed by GitHub
parent 96122b5b71
commit 3398744a61
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 428 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

322
lldb/docs/resources/debugging.rst
View File

@ -195,6 +195,11 @@ automatically debug the ``gdbserver`` process as it's created. However this
author has not been able to get either to work in this scenario so we suggest author has not been able to get either to work in this scenario so we suggest
making a more specific command wherever possible instead. making a more specific command wherever possible instead.
Another option is to let ``lldb-server`` start up, then attach to the process
that's interesting to you. It's less automated and won't work if the bug occurs
during startup. However it is a good way to know you've found the right one,
then you can take its command line and run that directly.
Output From ``lldb-server`` Output From ``lldb-server``
*************************** ***************************
@ -258,3 +263,320 @@ then ``lldb B`` to trigger ``lldb-server B`` to go into that code and hit the
breakpoint. ``lldb-server A`` is only here to let us debug ``lldb-server B`` breakpoint. ``lldb-server A`` is only here to let us debug ``lldb-server B``
remotely. remotely.
Debugging The Remote Protocol
-----------------------------
LLDB mostly follows the `GDB Remote Protocol <https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html>`_
. Where there are differences it tries to handle both LLDB and GDB behaviour.
LLDB does have extensions to the protocol which are documented in
`lldb-gdb-remote.txt <https://github.com/llvm/llvm-project/blob/main/lldb/docs/lldb-gdb-remote.txt>`_
and `lldb/docs/lldb-platform-packets.txt <https://github.com/llvm/llvm-project/blob/main/lldb/docs/lldb-platform-packets.txt>`_.
Logging Packets
***************
If you just want to observe packets, you can enable the ``gdb-remote packets``
log channel.
::
(lldb) log enable gdb-remote packets
(lldb) run
lldb < 1> send packet: +
lldb history[1] tid=0x264bfd < 1> send packet: +
lldb < 19> send packet: $QStartNoAckMode#b0
lldb < 1> read packet: +
You can do this on the ``lldb-server`` end as well by passing the option
``--log-channels "gdb-remote packets"``. Then you'll see both sides of the
connection.
Some packets may be printed in a nicer way than others. For example XML packets
will print the literal XML, some binary packets may be decoded. Others will just
be printed unmodified. So do check what format you expect, a common one is hex
encoded bytes.
You can enable this logging even when you are connecting to an ``lldb-server``
in platform mode, this protocol is used for that too.
Debugging Packet Exchanges
**************************
Say you want to make ``lldb`` send a packet to ``lldb-server``, then debug
how the latter builds its response. Maybe even see how ``lldb`` handles it once
it's sent back.
That all takes time, so LLDB will likely time out and think the remote has gone
away. You can change the ``plugin.process.gdb-remote.packet-timeout`` setting
to prevent this.
Here's an example, first we'll start an ``lldb-server`` being debugged by
``lldb``. Placing a breakpoint on a packet handler we know will be hit once
another ``lldb`` connects.
::
$ lldb -- lldb-server gdbserver :1234 -- /tmp/test.o
<...>
(lldb) b GDBRemoteCommunicationServerCommon::Handle_qSupported
Breakpoint 1: where = <...>
(lldb) run
<...>
Next we connect another ``lldb`` to this, with a timeout of 5 minutes:
::
$ lldb /tmp/test.o
<...>
(lldb) settings set plugin.process.gdb-remote.packet-timeout 300
(lldb) gdb-remote 1234
Doing so triggers the breakpoint in ``lldb-server``, bringing us back into
``lldb``. Now we've got 5 minutes to do whatever we need before LLDB decides
the connection has failed.
::
* thread #1, name = 'lldb-server', stop reason = breakpoint 1.1
frame #0: 0x0000aaaaaacc6848 lldb-server<...>
lldb-server`lldb_private::process_gdb_remote::GDBRemoteCommunicationServerCommon::Handle_qSupported:
-> 0xaaaaaacc6848 <+0>: sub sp, sp, #0xc0
<...>
(lldb)
Once you're done simply ``continue`` the ``lldb-server``. Back in the other
``lldb``, the connection process will continue as normal.
::
Process 2510266 stopped
* thread #1, name = 'test.o', stop reason = signal SIGSTOP
frame #0: 0x0000fffff7fcd100 ld-2.31.so`_start
ld-2.31.so`_start:
-> 0xfffff7fcd100 <+0>: mov x0, sp
<...>
(lldb)
Reducing Bugs
-------------
This section covers reducing a bug that happens in LLDB itself, or where you
suspect that LLDB causes something else to behave abnormally.
Since bugs vary wildly, the advice here is general and incomplete. Let your
instincts guide you and don't feel the need to try everything before reporting
an issue or asking for help. This is simply inspiration.
Reduction
*********
The first step is to reduce uneeded compexity where it is cheap to do so. If
something is easily removed or frozen to a cerain value, do so. The goal is to
keep the failure mode the same, with fewer dependencies.
This includes, but is not limited to:
* Removing test cases that don't crash.
* Replacing dynamic lookups with constant values.
* Replace supporting functions with stubs that do nothing.
* Moving the test case to less unqiue system. If your machine has an exotic
extension, try it on a readily available commodity machine.
* Removing irrelevant parts of the test program.
* Reproducing the issue without using the LLDB test runner.
* Converting a remote debuging scenario into a local one.
Now we hopefully have a smaller reproducer than we started with. Next we need to
find out what components of the software stack might be failing.
Some examples are listed below with suggestions for how to investigate them.
* Debugger
* Use a `released version of LLDB <https://github.com/llvm/llvm-project/releases>`_.
* If on MacOS, try the system ``lldb``.
* Try GDB or any other system debugger you might have e.g. Microsoft Visual
Studio.
* Kernel
* Start a virtual machine running a different version. ``qemu-system`` is
useful here.
* Try a different physical system running a different version.
* Remember that for most kernels, userspace crashing the kernel is always a
kernel bug. Even if the userspace program is doing something unconventional.
So it could be a bug in the application and the kernel.
* Compiler and compiler options
* Try other versions of the same compiler or your system compiler.
* Emit older versions of DWARF info, particularly DWARFv4 to v5, some tools
did/do not understand the new constructs.
* Reduce optimisation options as much as possible.
* Try all the language modes e.g. C++17/20 for C++.
* Link against LLVM's libcxx if you suspect a bug involving the system C++
library.
* For languages other than C/C++ e.g. Rust, try making an equivalent program
in C/C++. LLDB tends to try to fit other languages into a C/C++ mould, so
porting the program can make triage and reporting much easier.
* Operating system
* Use docker to try various versions of Linux.
* Use ``qemu-system`` to emulate other operating systems e.g. FreeBSD.
* Architecture
* Use `QEMU user space emulation <https://www.qemu.org/docs/master/user/main.html>`_
to quickly test other architectures. Note that ``lldb-server`` cannot be used
with this as the ptrace APIs are not emulated.
* If you need to test a big endian system use QEMU to emulate s390x (user
space emulation for just ``lldb``, ``qemu-system`` for testing
``lldb-server``).
.. note:: When using QEMU you may need to use the built in GDB stub, instead of
``lldb-server``. For example if you wanted to debug ``lldb`` running
inside ``qemu-user-s390x`` you would connect to the GDB stub provided
by QEMU.
The same applies if you want to see how ``lldb`` would debug a test
program that is running on s390x. It's not totally accurate because
you're not using ``lldb-server``, but this is fine for features that
are mostly implemented in ``lldb``.
If you are running a full system using ``qemu-system``, you likely
want to connect to the ``lldb-server`` running within the userspace
of that system.
If your test program is bare metal (meaning it requires no supporting
operating system) then connect to the built in GDB stub. This can be
useful when testing embedded systems or kernel debugging.
Reducing Ptrace Related Bugs
****************************
This section is written Linux specific but the same can likely be done on
other Unix or Unix like operating systems.
Sometimes you will find ``lldb-server`` doing something with ptrace that causes
a problem. Your reproducer involves running ``lldb`` as well, this is not going
to go over well with kernel and is generally more difficult to explain if you
want to get help with it.
If you think you can get your point across without this, no need. If you're
pretty sure you have for example found a Linux Kernel bug, doing this greatly
increases the chances it'll get fixed.
We'll remove the LLDB dependency by making a smaller standalone program that
does the same actions. Starting with a skeleton program that forks and debugs
the inferior process.
The program presented `here <https://eli.thegreenplace.net/2011/01/23/how-debuggers-work-part-1>`_
(`source <https://github.com/eliben/code-for-blog/blob/master/2011/simple_tracer.c>`_)
is a great starting point. There is also an AArch64 specific example in
`the LLDB examples folder <https://github.com/llvm/llvm-project/tree/main/lldb/examples/ptrace_example.c>`_.
For either, you'll need to modify that to fit your architecture. An tip for this
is to take any constants used in it, find in which function(s) they are used in
LLDB and then you'll find the equivalent constants in the same LLDB functions
for your architecture.
Once that is running as expected we can convert ``lldb-server``'s into calls in
this program. To get a log of those, run ``lldb-server`` with
``--log-channels "posix ptrace"``. You'll see output like:
::
$ lldb-server gdbserver :1234 --log-channels "posix ptrace" -- /tmp/test.o
1694099878.829990864 <...> ptrace(16896, 2659963, 0x0000000000000000, 0x000000000000007E, 0)=0x0
1694099878.830722332 <...> ptrace(16900, 2659963, 0x0000FFFFD14BF7CC, 0x0000FFFFD14BF7D0, 16)=0x0
1694099878.831967115 <...> ptrace(16900, 2659963, 0x0000FFFFD14BF66C, 0x0000FFFFD14BF630, 16)=0xffffffffffffffff
1694099878.831982136 <...> ptrace() failed: Invalid argument
Launched '/tmp/test.o' as process 2659963...
Each call is logged with its parameters and its result as the ``=`` on the end.
From here you will need to use a combination of the `ptrace documentation <https://man7.org/linux/man-pages/man2/ptrace.2.html>`_
and Linux Kernel headers (``uapi/linux/ptrace.h`` mainly) to figure out what
the calls are.
The most important parameter is the first, which is the request number. In the
example above ``16896``, which is hex ``0x4200``, is ``PTRACE_SETOPTIONS``.
Luckily, you don't usually have to figure out all those early calls. Our
skeleton program will be doing all that, successfully we hope.
What you should do is record just the interesting bit to you. Let's say
something odd is happening when you read the ``tpidr`` register (this is an
AArch64 register, just for example purposes).
First, go to the ``lldb-server`` terminal and press enter a few times to put
some blank lines after the last logging output.
Then go to your ``lldb`` and:
::
(lldb) register read tpidr
tpidr = 0x0000fffff7fef320
You'll see this from ``lldb-server``:
::
<...> ptrace(16900, 2659963, 0x0000FFFFD14BF6CC, 0x0000FFFFD14BF710, 8)=0x0
If you don't see that, it may be because ``lldb`` has cached it. The easiest way
to clear that cache is to step. Remember that some registers are read every
step, so you'll have to adjust depending on the situation.
Assuming you've got that line, you would look up what ``116900`` is. This is
``0x4204`` in hex, which is ``PTRACE_GETREGSET``. As we expected.
The following parameters are not as we might expect because what we log is a bit
different from the literal ptrace call. See your platform's definition of
``PtraceWrapper`` for the exact form.
The point of all this is that by doing a single action you can get a few
isolated ptrace calls and you can then fill in the blanks and write
equivalent calls in the skeleton program.
The final piece of this is likely breakpoints. Assuming your bug does not
require a hardware breakpoint, you can get software breakpoints by inserting
a break instruction into the inferior's code at compile time. Usually by using
an architecture specific assembly statement, as you will need to know exactly
how many instructions to overwrite later.
Doing it this way instead of exactly copying what LLDB does will save a few
ptrace calls. The AArch64 example program shows how to do this.
* The inferior contains ``BRK #0`` then ``NOP``.
* 2 4 byte instructins means 8 bytes of data to replace, which matches the
minimum size you can write with ``PTRACE_POKETEXT``.
* The inferior runs to the ``BRK``, which brings us into the debugger.
* The debugger reads ``PC`` and writes ``NOP`` then ``NOP`` to the location
pointed to by ``PC``.
* The debugger then single steps the inferior to the next instruction
(this is not required in this specific scenario, you could just continue but
it is included because this more cloesly matches what ``lldb`` does).
* The debugger then continues the inferior.
* The inferior exits, and the whole program exits.
Using this technique you can emulate the usual "run to main, do a thing" type
reproduction steps.
Finally, that "thing" is the ptrace calls you got from the ``lldb-server`` logs.
Add those to the debugger function and you now have a reproducer that doesn't
need any part of LLDB.

106
lldb/examples/ptrace_example.c Normal file
View File

@ -0,0 +1,106 @@
//===-- ptrace_example.c --------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include <asm/ptrace.h>
#include <linux/elf.h>
#include <stdint.h>
#include <stdio.h>
#include <sys/prctl.h>
#include <sys/ptrace.h>
#include <sys/uio.h>
#include <sys/wait.h>
#include <unistd.h>
// The demo program shows how to do basic ptrace operations without lldb
// or lldb-server. For the purposes of experimentation or reporting bugs
// in kernels.
//
// It is AArch64 Linux specific, adapt as needed.
//
// Expected output:
// Before breakpoint
// After breakpoint
void inferior() {
if (ptrace(PTRACE_TRACEME, 0, 0, 0) < 0) {
perror("ptrace");
return;
}
printf("Before breakpoint\n");
// Go into debugger. Instruction replaced with nop later.
// We write 2 instuctions because POKETEXT works with
// 64 bit values and we don't want to overwrite the
// call to printf accidentally.
asm volatile("BRK #0 \n nop");
printf("After breakpoint\n");
}
void debugger(pid_t child) {
int wait_status;
// Wait until it hits the breakpoint.
wait(&wait_status);
while (WIFSTOPPED(wait_status)) {
if (WIFEXITED(wait_status)) {
printf("inferior exited normally\n");
return;
}
// Read general purpose registers to find the PC value.
struct user_pt_regs regs;
struct iovec io;
io.iov_base = &regs;
io.iov_len = sizeof(regs);
if (ptrace(PTRACE_GETREGSET, child, NT_PRSTATUS, &io) < 0) {
printf("getregset failed\n");
return;
}
// Replace brk #0 / nop with nop / nop by writing to memory
// at the current PC.
uint64_t replace = 0xd503201fd503201f;
if (ptrace(PTRACE_POKETEXT, child, regs.pc, replace) < 0) {
printf("replacing bkpt failed\n");
return;
}
// Single step over where the brk was.
if (ptrace(PTRACE_SINGLESTEP, child, 0, 0) < 0) {
perror("ptrace");
return;
}
// Wait for single step to be done.
wait(&wait_status);
// Run to completion.
if (ptrace(PTRACE_CONT, child, 0, 0) < 0) {
perror("ptrace");
return;
}
// Wait to see that the inferior exited.
wait(&wait_status);
}
}
int main() {
pid_t child = fork();
if (child == 0)
inferior();
else if (child > 0)
debugger(child);
else
return -1;
return 0;
}