Similar to `__asan_default_options`, users can specify default options
upon building the instrumented binaries by providing their own
definition of `__xray_default_options` which returns the option strings.
This is useful in cases where setting the `XRAY_OPTIONS` environment
variable might be difficult. Plus, it's a convenient way to populate
XRay options when you always want the instrumentation to be enabled.
This patch implements support for `-fxray-shared` on AArch64 and fixes a
remaining issue in the previous PR #114431.
A bug in the XRay `CMakeLists.txt` caused the XRay assembly sources to
be built for every architecture in `XRAY_DSO_SUPPORTED_ARCH` on Apple.
This led to the compiler trying to compile AArch64 assembly for X86
targets and vice versa.
This is addressed here by ensuring that assembly sources are only built
for the matching architecture (see fixup commit).
**Original PR description:**
This patch adds support for `-fxray-shared` on AArch64. This feature,
introduced in #113548 for x86_64, enables the instrumentation of shared
libraries with XRay.
Changes:
- Adds AArch64 to the list of targets supporting `-fxray-shared`
- Introduces PIC versions of the AArch64 XRay trampolines
- Adjusts relevant XRay tests
This patch adds support for `-fxray-shared` on AArch64. This feature,
introduced in #113548 for x86_64, enables the instrumentation of shared
libraries with XRay.
Changes:
- Adds AArch64 to the list of targets supporting `-fxray-shared`
- Introduces PIC versions of the AArch64 XRay trampolines
- Adjusts relevant XRay tests
Compiling with `-fxray-shared` requires position-independent code
(introduced in #113548).
Some tests do not explicitly specify this, thus falling back to the
compiler default.
If, for example, Clang is compiled with
`-DCLANG_DEFAULT_PIE_ON_LINUX=OFF`, these checks fail.
This patch addresses this issue in two tests:
- Removing a check in `xray-shared.cpp` that only tests default PIC
behavior
- Adding `-fPIC` explicitly in `clang-xray-shared.cpp`
This fixes remaining issues in my previous PR #90959.
Changes:
- Removed dependency on LLVM header in `xray_interface.cpp`
- Fixed XRay patching for some targets due to missing changes in
architecture-specific patching functions
- Addressed some remaining compiler warnings that I missed in the
previous patch
- Formatting
I have tested these changes on `x86_64` (natively), as well as
`ppc64le`, `aarch64` and `arm32` (cross-compiled and emulated using
qemu).
**Original description:**
This PR introduces shared library (DSO) support for XRay based on a
revised version of the implementation outlined in [this
RFC](https://discourse.llvm.org/t/rfc-upstreaming-dso-instrumentation-support-for-xray/73000).
The feature enables the patching and handling of events from DSOs,
supporting both libraries linked at startup or explicitly loaded, e.g.
via `dlopen`.
This patch adds the following:
- The `-fxray-shared` flag to enable the feature (turned off by default)
- A small runtime library that is linked into every instrumented DSO,
providing position-independent trampolines and code to register with the
main XRay runtime
- Changes to the XRay runtime to support management and patching of
multiple objects
These changes are fully backward compatible, i.e. running without
instrumented DSOs will produce identical traces (in terms of recorded
function IDs) to the previous implementation.
Due to my limited ability to test on other architectures, this feature
is only implemented and tested with x86_64. Extending support to other
architectures is fairly straightforward, requiring only a
position-independent implementation of the architecture-specific
trampoline implementation (see
`compiler-rt/lib/xray/xray_trampoline_x86_64.S` for reference).
This patch does not include any functionality to resolve function IDs
from DSOs for the provided logging/tracing modes. These modes still work
and will record calls from DSOs, but symbol resolution for these
functions in not available. Getting this to work properly requires
recording information about the loaded DSOs and should IMO be discussed
in a separate RFC, as there are mulitple feasible approaches.
---------
Co-authored-by: Sebastian Kreutzer <sebastian.kreutzer@tu-darmstadt.de>
This PR introduces shared library (DSO) support for XRay based on a
revised version of the implementation outlined in [this
RFC](https://discourse.llvm.org/t/rfc-upstreaming-dso-instrumentation-support-for-xray/73000).
The feature enables the patching and handling of events from DSOs,
supporting both libraries linked at startup or explicitly loaded, e.g.
via `dlopen`.
This patch adds the following:
- The `-fxray-shared` flag to enable the feature (turned off by default)
- A small runtime library that is linked into every instrumented DSO,
providing position-independent trampolines and code to register with the
main XRay runtime
- Changes to the XRay runtime to support management and patching of
multiple objects
These changes are fully backward compatible, i.e. running without
instrumented DSOs will produce identical traces (in terms of recorded
function IDs) to the previous implementation.
Due to my limited ability to test on other architectures, this feature
is only implemented and tested with x86_64. Extending support to other
architectures is fairly straightforward, requiring only a
position-independent implementation of the architecture-specific
trampoline implementation (see
`compiler-rt/lib/xray/xray_trampoline_x86_64.S` for reference).
This patch does not include any functionality to resolve function IDs
from DSOs for the provided logging/tracing modes. These modes still work
and will record calls from DSOs, but symbol resolution for these
functions in not available. Getting this to work properly requires
recording information about the loaded DSOs and should IMO be discussed
in a separate RFC, as there are mulitple feasible approaches.
@petrhosek @jplehr
When running tests in compiler-rt using the lit internal shell with the
following command:
```
LIT_USE_INTERNAL_SHELL=1 ninja check-compiler-rt
```
one common error that occurs is:
```
'XRAY_OPTIONS=patch_premain=false:verbosity=1': command not found
```
This error, along with over 50 similar "environment variable not found"
errors, appears across 35 files in complier-rt. These errors happen
because the environment variables are not being set correctly when the
internal shell is used, leading to commands failing due to unrecognized
environment variables.
This patch addresses the issue by using the `env` command to properly
set the environment variables before running the tests. By explicitly
setting the environment variables through `env`, the internal shell can
correctly interpret and apply them, allowing the tests to pass.
fixes: #102395
[link to
RFC](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
They are checked in the parent CMakeLists in
`compiler_rt_build_runtime` and `compiler_rt_test_runtime`.
There are non-redundant checks when a sanitizer checks
for the presense of another one. They should not be removed.
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
With https://github.com/llvm/llvm-project/pull/83088, we now need the
runtimes to be built before running test if
COMPILER_RT_TEST_STANDALONE_BUILD_LIBS is true, since otherwise we
get failures running `ninja check-all` such as the following:
```
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.fuzzer-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-basic-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-fdr-x86_64.a: No such file or directory
```
This is a follow-up to 058e9b03 which started removing these checks
and it should make it easier to stop forcing COMPILER_RT_STANDALONE_BUILD
for runtimes builds in the future.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83651
fdr-thread-order.cpp can be very slow when the thread contention is large.
Enable it for AArch64 and x86-64 for now.
fdr-mode.cpp fails on a ppc64le machine. Unsupport it on ppc64le for now.
The remaining modified tests pass on AArch64, ppc64le, and x86-64.
`__xray_customevent` and `__xray_typedevent` are built-in functions in Clang.
With -fxray-instrument, they are lowered to intrinsics llvm.xray.customevent and
llvm.xray.typedevent, respectively. These intrinsics are then lowered to
TargetOpcode::{PATCHABLE_EVENT_CALL,PATCHABLE_TYPED_EVENT_CALL}. The target is
responsible for generating a code sequence that calls either
`__xray_CustomEvent` (with 2 arguments) or `__xray_TypedEvent` (with 3
arguments).
Before patching, the code sequence is prefixed by a branch instruction that
skips the rest of the code sequence. After patching
(compiler-rt/lib/xray/xray_AArch64.cpp), the branch instruction becomes a NOP
and the function call will take effects.
This patch implements the lowering process for
{PATCHABLE_EVENT_CALL,PATCHABLE_TYPED_EVENT_CALL} and implements the runtime.
```
// Lowering of PATCHABLE_EVENT_CALL
.Lxray_sled_N:
b #24
stp x0, x1, [sp, #-16]!
x0 = reg of op0
x1 = reg of op1
bl __xray_CustomEvent
ldrp x0, x1, [sp], #16
```
As a result, two updated tests in compiler-rt/test/xray/TestCases/Posix/ now
pass on AArch64.
Reviewed By: peter.smith
Differential Revision: https://reviews.llvm.org/D153320
The Clang built-in function is void __xray_typedevent(size_t, const void *, size_t),
but the LLVM intrinsics has smaller integer types. Since we only allow
64-bit ELF/Mach-O targets, we can change llvm.xray.typedevent to
i64/ptr/i64.
This allows encoding more information and avoids i16 legalization for
many non-X86 targets.
fdrLoggingHandleTypedEvent only supports uint16_t event type.
This is an ongoing series of commits that are reformatting our
Python code. This catches the last of the python files to
reformat. Since they where so few I bunched them together.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: jhenderson, #libc, Mordante, sivachandra
Differential Revision: https://reviews.llvm.org/D150784
We run bots on a shared machine and under high load
this test sometimes segfaults.
https://lab.llvm.org/buildbot/#/builders/185/builds/1368
==1952234==XRay FDR init successful.
==1952234==XRay FDR: Not flushing to file, 'no_file_flush=true'.
<...>fdr-reinit.cpp.script: line 4: 1952234 Segmentation fault
XRAY_OPTIONS="verbosity=1" <...>/fdr-reinit.cpp.tmp
Looking at the printed output I think it's happening at:
// Finally, we should signal the sibling thread to stop.
keep_going.clear(std::memory_order_release);
Disabling the test while I try to reproduce.
Summary:
Add a flag to omit the xray_fn_idx to cut size overhead and relocations
roughly in half at the cost of reduced performance for single function
patching. Minor additions to compiler-rt support per-function patching
without the index.
Reviewers: dberris, MaskRay, johnislarry
Subscribers: hiraditya, arphaman, cfe-commits, #sanitizers, llvm-commits
Tags: #clang, #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D81995
Summary:
Because the dynamic linker for 32-bit executables on 64-bit FreeBSD uses
the environment variable `LD_32_LIBRARY_PATH` instead of
`LD_LIBRARY_PATH` to find needed dynamic libraries, running the 32-bit
parts of the dynamic ASan tests will fail with errors similar to:
```
ld-elf32.so.1: Shared object "libclang_rt.asan-i386.so" not found, required by "Asan-i386-inline-Dynamic-Test"
```
This adds support for setting up `LD_32_LIBRARY_PATH` for the unit and
regression tests. It will likely also require a minor change to the
`TestingConfig` class in `llvm/utils/lit/lit`.
Reviewers: emaste, kcc, rnk, arichardson
Reviewed By: arichardson
Subscribers: kubamracek, krytarowski, fedor.sergeev, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65772
llvm-svn: 368516
These lit configuration files are really Python source code. Using the
.py file extension helps editors and tools use the correct language
mode. LLVM and Clang already use this convention for lit configuration,
this change simply applies it to all of compiler-rt.
Reviewers: vitalybuka, dberris
Differential Revision: https://reviews.llvm.org/D63658
llvm-svn: 364591
Summary: When libc++ is used to build CLANG, its XRay libraries libclang_rt.xray-*.a have dependencies on libc++. Therefore, libc++ is needed to link and run XRay test cases. For Linux -rpath is also needed to specify where to load libc++. This change sets macro LLVM_LIBCXX_USED to 1 if libc++ is actually used in the build. XRay tests then check the flag and add -L<llvm_shlib_dir> -lc++ and -Wl,-rpath=<llvm_shlib_dir> if needed.
Reviewers: hubert.reinterpretcast, amyk, dberris, jasonliu, sfertile, EricWF
Subscribers: dberris, mgorny, jsji, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61016
llvm-svn: 360060
Summary:
This change cuts across LLVM and compiler-rt to add support for
rendering custom events in the XRayRecord type, to allow for including
user-provided annotations in the output YAML (as raw bytes).
This work enables us to add custom event and typed event records into
the `llvm::xray::Trace` type for user-provided events. This can then be
programmatically handled through the C++ API and can be included in some
of the tooling as well. For now we support printing the raw data we
encounter in the custom events in the converted output.
Future work will allow us to start interpreting these custom and typed
events through a yet-to-be-defined API for extending the trace analysis
library.
Reviewers: mboerger
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D54139
llvm-svn: 346214
Summary:
Currently,
cd test/xray/TestCases/Posix
$build/bin/llvm-lit fdr-thread-order.cc
fails because `rm fdr-thread-order.*` deletes the .cc file.
This patch uses:
* %t as temporary directory name containing log files
* %t.exe as executable name
It does not delete %t after the test finishes for debugging convenience.
This matches the behavior of tests of various other LLVM components.
Log files will not clog up because the temporary directory (unique among
test files but the same among multiple invocations of a test) is cleaned
at the beginning of the test.
Reviewers: dberris, mboerger, eizan
Reviewed By: dberris
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D52638
llvm-svn: 343295
Summary:
This change allows us to use the library path from which the LLVM
libraries are installed, in case the LLVM installation generates shared
libraries.
This should address llvm.org/PR39070.
Reviewers: mboerger, eizan
Subscribers: mgorny, jfb, llvm-commits
Differential Revision: https://reviews.llvm.org/D52597
llvm-svn: 343280
This API has been deprecated three months ago and shouldn't be used
anymore, all clients should migrate to the new string based API.
Differential Revision: https://reviews.llvm.org/D51606
llvm-svn: 342318
Summary:
Before this change, we only emit the XRay attributes in LLVM IR when the
-fxray-instrument flag is provided. This may cause issues with thinlto
when the final binary is being built/linked with -fxray-instrument, and
the constitutent LLVM IR gets re-lowered with xray instrumentation.
With this change, we can honour the "never-instrument "attributes
provided in the source code and preserve those in the IR. This way, even
in thinlto builds, we retain the attributes which say whether functions
should never be XRay instrumented.
This change addresses llvm.org/PR38922.
Reviewers: mboerger, eizan
Subscribers: mehdi_amini, dexonsmith, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D52015
llvm-svn: 342200
Summary:
This change provides access to the file header even in the in-memory
buffer processing. This allows in-memory processing of the buffers to
also check the version, and the format, of the profile data.
Reviewers: eizan, kpw
Reviewed By: eizan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D50037
llvm-svn: 338347
By default, xray filters events that takes less than 5uS from its log.
In this existing test, should printf complete very quickly this will
lead to test-critical function calls being filtered (i.e. print_parent_tid).
Given that we're not testing the filtering feature, disable it for this
test.
llvm-svn: 337929
