109 Commits

Author SHA1 Message Date
Marc Auberer
cac7821438
[compiler-rt] Replace deprecated aligned_storage with aligned byte array (#94171)
`std::aligned_storage` is deprecated with C++23, see
[here](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1413r3.pdf).

This replaces the usages of `std::aligned_storage` within compiler-rt
with an aligned `std::byte` array.
I will provide patches for other subcomponents as well.
2024-06-08 12:29:01 +02:00
Michael Kruse
a35ac42fac
[compiler-rt] Revise IDE folder structure (#89753)
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.

 * Ensure that every target is in a folder
 * Use a folder hierarchy with each LLVM subproject as a top-level folder
 * Use consistent folder names between subprojects
 * When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
2024-06-04 09:26:45 +02:00
Aaron Siddhartha Mondal
852aaf5407
Reapply "[Support] Remove terminfo dependency (#92865)" (#93889)
This reverts commit fe82a3da36196157c0caa1ef2505186782f750d1.

This broke LLDB on MacOS due to a missing symbol during linking.

The fix has been applied in c6c08eee37bada190bd1aa4593c88a5e2c8cdaac.

Original commit message:

The terminfo dependency introduces a significant nonhermeticity into the
build. It doesn't respect `--no-undefined-version` meaning that it's not
a dependency that can be built with Clang 17+. This forces maintainers
of source-based distributions to implement patches or ignore linker
errors.

Remove it to reduce the closure size and improve portability of
LLVM-based tools. Users can still use command line arguments to toggle
color support expliticly.

Fixes #75490
Closes #53294 #23355
2024-05-31 01:29:00 +02:00
Michael Buch
fe82a3da36 Revert "[Support] Remove terminfo dependency (#92865)"
This reverts commit 6bf450c7a60fa62c642e39836566da94bb9bbc91.

It breaks LLDB CI: https://green.lab.llvm.org/job/llvm.org/view/LLDB/job/as-lldb-cmake/4762/execution/node/97/log/

```
/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -Wdocumentation -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -Wno-deprecated-declarations -Wno-unknown-pragmas -Wno-strict-aliasing -Wno-deprecated-register -Wno-vla-extension -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -mmacosx-version-min=14.1 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-dead_strip -Wl,-no_warn_duplicate_libraries tools/lldb/unittests/Editline/CMakeFiles/EditlineTests.dir/EditlineTest.cpp.o -o tools/lldb/unittests/Editline/EditlineTests  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/liblldbHost.a  lib/liblldbUtility.a  lib/libLLVMTestingSupport.a  /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libxml2.tbd  /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libedit.tbd  lib/liblldbHostMacOSXObjCXX.a  lib/liblldbUtility.a  -framework Foundation  -framework CoreFoundation  -framework CoreServices  -framework Security  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMBitReader.a  lib/libLLVMAsmParser.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMTextAPI.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libllvm_gtest.a  lib/libLLVMSupport.a  -lm  /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/lib/libz.tbd  /opt/homebrew/lib/libzstd.dylib  lib/libLLVMDemangle.a  -lpthread && cd /Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/lldb-build/tools/lldb/unittests/Editline && /opt/homebrew/Cellar/cmake/3.28.3/bin/cmake -E make_directory /Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/lldb-build/tools/lldb/unittests/Editline/./Inputs
ld: Undefined symbols:
  _setupterm, referenced from:
      lldb_private::Editline::Editline(char const*, __sFILE*, __sFILE*, __sFILE*, std::__1::recursive_mutex&) in liblldbHost.a[35](Editline.cpp.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
2024-05-29 16:20:42 +01:00
Aaron Siddhartha Mondal
6bf450c7a6
[Support] Remove terminfo dependency (#92865)
The terminfo dependency introduces a significant nonhermeticity into the
build. It doesn't respect `--no-undefined-version` meaning that it's not
a dependency that can be built with Clang 17+. This forces maintainers
of source-based distributions to implement patches or ignore linker
errors.

Remove it to reduce the closure size and improve portability of
LLVM-based tools. Users can still use command line arguments to toggle
color support expliticly.

Fixes #75490
Closes #53294 #23355
2024-05-24 20:20:15 +02:00
Alexander Richardson
55b90b5140
[compiler-rt] Remove llvm_gtest dependency from unit tests
All these unit tests already include ${COMPILER_RT_GTEST_SOURCE} as an
input source file and the target llvm_gtest does not exist for
standalone builds. Currently the DEPS argument is ignored for standalone
builds so the missing target is not a problem, but as part of fixing a
build race for standalone builds I am planning to include those
dependencies in COMPILER_RT_TEST_STANDALONE_BUILD_LIBS configurations.

Reviewed By: vitalybuka

Pull Request: https://github.com/llvm/llvm-project/pull/83649
2024-03-13 11:28:44 -07:00
Michał Górny
23c665371a [compiler-rt] Switch from llvm-config to find_package(LLVM)
Replace the use of the deprecated `llvm-config` tool with LLVM's CMake
files for detecting LLVM in standalone builds.

Differential Revision: https://reviews.llvm.org/D137024
2022-11-02 07:17:42 +01:00
Michał Górny
a05810e8f2 Revert "[compiler-rt] Switch from llvm-config to find_package(LLVM)"
This reverts commit 37acf9bdd4686397a6d37000e8d3fe35f456a860.  It broke
a buildbot.
2022-11-01 05:49:50 +01:00
Michał Górny
37acf9bdd4 [compiler-rt] Switch from llvm-config to find_package(LLVM)
Replace the use of the deprecated `llvm-config` tool with LLVM's CMake
files for detecting LLVM in standalone builds.

Differential Revision: https://reviews.llvm.org/D137024
2022-11-01 04:19:31 +01:00
Petr Hosek
6699f55488 [compiler-rt][CMake] Set --unwindlib=none when using LLVM libunwind
We already link libunwind explicitly so avoid trying to link toolchain's
default libunwind which may be missing. This matches what we already do
for libcxx and libcxxabi.

Differential Revision: https://reviews.llvm.org/D129472
2022-07-15 07:25:38 +00:00
Petr Hosek
ba007f20bb [CMake] Use explicit header path when using in-tree libc++ for tests
This is a follow up to D118200 which applies a similar cleanup to
headers when using in-tree libc++ to avoid accidentally picking up
the system headers.

Differential Revision: https://reviews.llvm.org/D128035
2022-07-09 06:14:29 +00:00
Stella Laurenzo
38151a08c2 Reapply "[cmake] Prefix gtest and gtest_main with "llvm_"."
This reverts commit 7cdda6b8ce49ae3c90c068cff4dc355bba5d77f2.

Differential Revision: https://reviews.llvm.org/D121020
2022-03-04 13:45:43 -08:00
Stella Laurenzo
7cdda6b8ce Revert "[cmake] Prefix gtest and gtest_main with "llvm_"."
lldb buildbot failure. will investigate and roll forward.

This reverts commit 9f37775472b45986b0ecce5243bd6ce119e5bd69.
2022-03-02 11:13:46 -08:00
Stella Laurenzo
9f37775472 [cmake] Prefix gtest and gtest_main with "llvm_".
The upstream project ships CMake rules for building vanilla gtest/gmock which conflict with the names chosen by LLVM. Since LLVM's build rules here are quite specific to LLVM, prefixing them to avoid collision is the right thing (i.e. there does not appear to be a path to letting someone *replace* LLVM's googletest with one they bring, so co-existence should be the goal).

This allows LLVM to be included with testing enabled within projects that themselves have a dependency on an official gtest release.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D120789
2022-03-02 10:53:32 -08:00
Petr Hosek
48a38954c9 [CMake] Use generator expression to get in-tree libc++ path
When using the in-tree libc++, we should be using the full path to
ensure that we're using the right library and not accidentally pick up
the system library.

Differential Revision: https://reviews.llvm.org/D118200
2022-01-26 14:12:48 -08:00
Kazuaki Ishizaki
a1e7e401d2 [compiler-rt] NFC: Fix trivial typo
Reviewed By: xgupta

Differential Revision: https://reviews.llvm.org/D77457
2021-09-04 14:12:58 +05:30
Petr Hosek
3c7bfbd683 [CMake] Use find_library for ncurses
Currently it is hard to avoid having LLVM link to the system install of
ncurses, since it uses check_library_exists to find e.g. libtinfo and
not find_library or find_package.

With this change the ncurses lib is found with find_library, which also
considers CMAKE_PREFIX_PATH. This solves an issue for the spack package
manager, where we want to use the zlib installed by spack, and spack
provides the CMAKE_PREFIX_PATH for it.

This is a similar change as https://reviews.llvm.org/D79219, which just
landed in master.

Patch By: haampie

Differential Revision: https://reviews.llvm.org/D85820
2020-08-31 20:06:21 -07:00
Harmen Stoppels
cdcb9ab10e Revert "Use find_library for ncurses"
The introduction of find_library for ncurses caused more issues than it solved problems. The current open issue is it makes the static build of LLVM fail. It is better to revert for now, and get back to it later.

Revert "[CMake] Fix an issue where get_system_libname creates an empty regex capture on windows"
This reverts commit 1ed1e16ab83f55d85c90ae43a05cbe08a00c20e0.

Revert "Fix msan build"
This reverts commit 34fe9613dda3c7d8665b609136a8c12deb122382.

Revert "[CMake] Always mark terminfo as unavailable on Windows"
This reverts commit 76bf26236f6fd453343666c3cd91de8f74ffd89d.

Revert "[CMake] Fix OCaml build failure because of absolute path in system libs"
This reverts commit 8e4acb82f71ad4effec8895b8fc957189ce95933.

Revert "[CMake] Don't look for terminfo libs when LLVM_ENABLE_TERMINFO=OFF"
This reverts commit 495f91fd33d492941c39424a32cf24bcfe192f35.

Revert "Use find_library for ncurses"
This reverts commit a52173a3e56553d7b795bcf3cdadcf6433117107.

Differential revision: https://reviews.llvm.org/D86521
2020-08-27 17:57:26 -07:00
Harmen Stoppels
a52173a3e5 Use find_library for ncurses
Currently it is hard to avoid having LLVM link to the system install of
ncurses, since it uses check_library_exists to find e.g. libtinfo and
not find_library or find_package.

With this change the ncurses lib is found with find_library, which also
considers CMAKE_PREFIX_PATH. This solves an issue for the spack package
manager, where we want to use the zlib installed by spack, and spack
provides the CMAKE_PREFIX_PATH for it.

This is a similar change as https://reviews.llvm.org/D79219, which just
landed in master.

Differential revision: https://reviews.llvm.org/D85820
2020-08-17 19:52:52 -07:00
Jonas Devlieghere
8903433bb2 Re-land "[compiler-rt] Migrate llvm::make_unique to std::make_unique"
With the compiler-rt check for C++14 updated in r368960, this should now
be fine to land.

llvm-svn: 369009
2019-08-15 14:57:44 +00:00
Jonas Devlieghere
d65f37dbcc Revert "[compiler-rt] Migrate llvm::make_unique to std::make_unique"
The X-ray unit tests in compiler-rt are overriding the C++ version by
explicitly passing -std=c++11 in the compiler invocation. This poses a
problem as these tests are including LLVM headers that can now use C++14
features. I'm temporarily reverting this as I investigate the correct
solution.

llvm-svn: 368952
2019-08-15 03:02:57 +00:00
Jonas Devlieghere
5edd6843fa [compiler-rt] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368946
2019-08-15 00:06:49 +00:00
Nico Weber
bc0d697db9 compiler-rt: Rename .cc file in lib/xray/tests/unit to .cpp
Like r367463, but for xray/texts/unit.

llvm-svn: 367550
2019-08-01 12:35:27 +00:00
Hubert Tong
50721b27c4 [compiler-rt][tests] Propagate COMPILER_RT_UNITTEST_LINK_FLAGS
`COMPILER_RT_UNITTEST_LINK_FLAGS` is dropped in many places, unlike
`COMPILER_RT_UNITTEST_CFLAGS`. This patch attempts to remove that
inconsistency.

Previously reviewed as part of D58951.

Reviewers: sfertile, peter.smith, pzheng, phosek, Hahnfeld, nemanjai, jasonliu

Reviewed By: sfertile

Subscribers: jsji, kubamracek, dberris, mgorny, delcypher, jdoerfert, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D60143

llvm-svn: 359733
2019-05-01 22:25:16 +00:00
Jonas Hahnfeld
a05d442391 [compiler-rt] Cleanup usage of C++ ABI library
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.

Recommit r354132 which I reverted in r354153 because it broke a sanitizer
bot. This was because of the "fixes" for pthread linking, so I've removed
these changes.

Differential Revision: https://reviews.llvm.org/D58012

llvm-svn: 354198
2019-02-16 08:34:26 +00:00
Jonas Hahnfeld
ea686e2845 Revert "[compiler-rt] Cleanup usage of C++ ABI library"
This reverts r354132 because it breaks sanitizer-x86_64-linux:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/19915

llvm-svn: 354153
2019-02-15 18:25:26 +00:00
Jonas Hahnfeld
3cc63cfaa7 [compiler-rt] Cleanup usage of C++ ABI library
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.

Differential Revision: https://reviews.llvm.org/D58012

llvm-svn: 354132
2019-02-15 14:30:18 +00:00
Hubert Tong
a9f91c810c [xray][tests][RHDTS] Add -lstdc++ after LLVM libs, resolving link error with RHDTS
Summary:
A link error was encountered when using the Red Hat Developer Toolset.
In the RHDTS, `libstdc++.so` is a linker script that may resolve symbols
to a static library. This patch places `-lstdc++` later in the ordering.

Reviewers: sfertile, nemanjai, tstellar, dberris

Reviewed By: dberris

Subscribers: dberris, mgorny, delcypher, jdoerfert, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D58144

llvm-svn: 353905
2019-02-13 00:55:30 +00:00
Dimitry Andric
d849f8fd8f Fix XRayTest link on FreeBSD (and likely NetBSD too)
Summary:
As reported on llvm-testers, during 8.0.0-rc1 testing I got errors while
building of `XRayTest`, during `check-all`:

```
[100%] Generating XRayTest-x86_64-Test
/home/dim/llvm/8.0.0/rc1/Phase3/Release/llvmCore-8.0.0-rc1.obj/./lib/libLLVMSupport.a(Signals.cpp.o): In function `llvm::sys::PrintStackTrace(llvm::raw_ostream&)':
Signals.cpp:(.text._ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x24): undefined reference to `backtrace'
Signals.cpp:(.text._ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x254): undefined reference to `llvm::itaniumDemangle(char const*, char*, unsigned long*, int*)'
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [projects/compiler-rt/lib/xray/tests/unit/CMakeFiles/TXRayTest-x86_64-Test.dir/build.make:73: projects/compiler-rt/lib/xray/tests/unit/XRayTest-x86_64-Test] Error 1
gmake[3]: Target 'projects/compiler-rt/lib/xray/tests/unit/CMakeFiles/TXRayTest-x86_64-Test.dir/build' not remade because of errors.
gmake[2]: *** [CMakeFiles/Makefile2:33513: projects/compiler-rt/lib/xray/tests/unit/CMakeFiles/TXRayTest-x86_64-Test.dir/all] Error 2
gmake[2]: Target 'CMakeFiles/check-all.dir/all' not remade because of errors.
gmake[1]: *** [CMakeFiles/Makefile2:737: CMakeFiles/check-all.dir/rule] Error 2
gmake[1]: Target 'check-all' not remade because of errors.
gmake: *** [Makefile:277: check-all] Error 2
[Release Phase3] check-all failed
```

This is because the `backtrace` function requires `-lexecinfo` on BSD
platforms.  To fix this, detect the `execinfo` library in
`cmake/config-ix.cmake`, and add it to the unit test link flags.

Additionally, since the code in `sys::PrintStackTrace` makes use of
`itaniumDemangle`, also add `-lLLVMDemangle`.  (Note that this is more
of a general problem with libLLVMSupport, but I'm looking for a quick
fix now so it can be merged to the 8.0 branch.)

Reviewers: dberris, hans, mgorny, samsonov

Reviewed By: dberris

Subscribers: krytarowski, delcypher, erik.pilkington, #sanitizers, emaste, llvm-commits

Differential Revision: https://reviews.llvm.org/D57181

llvm-svn: 352234
2019-01-25 19:36:47 +00:00
Chandler Carruth
2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Michal Gorny
1e8e4fc834 [xray] [tests] Detect and handle missing LLVMTestingSupport gracefully
Add a code to properly test for presence of LLVMTestingSupport library
when performing a stand-alone build, and skip tests requiring it when
it is not present.  Since the library is not installed, llvm-config
reported empty --libs for it and the tests failed to link with undefined
references.  Skipping the two fdr_* test files is better than failing to
build, and should be good enough until we find a better solution.

NB: both installing LLVMTestingSupport and building it automatically
from within compiler-rt sources are non-trivial.  The former due to
dependency on gtest, the latter due to tight integration with LLVM
source tree.

Differential Revision: https://reviews.llvm.org/D55891

llvm-svn: 349899
2018-12-21 13:37:30 +00:00
Dean Michael Berris
25d505953a [XRay] Use preallocated memory for XRay profiling
Summary:
This change builds upon D54989, which removes memory allocation from the
critical path of the profiling implementation. This also changes the API
for the profile collection service, to take ownership of the memory and
associated data structures per-thread.

The consolidation of the memory allocation allows us to do two things:

- Limits the amount of memory used by the profiling implementation,
  associating preallocated buffers instead of allocating memory
  on-demand.

- Consolidate the memory initialisation and cleanup by relying on the
  buffer queue's reference counting implementation.

We find a number of places which also display some problematic
behaviour, including:

- Off-by-factor bug in the allocator implementation.

- Unrolling semantics in cases of "memory exhausted" situations, when
  managing the state of the function call trie.

We also add a few test cases which verify our understanding of the
behaviour of the system, with important edge-cases (especially for
memory-exhausted cases) in the segmented array and profile collector
unit tests.

Depends on D54989.

Reviewers: mboerger

Subscribers: dschuff, mgorny, dmgreen, jfb, llvm-commits

Differential Revision: https://reviews.llvm.org/D55249

llvm-svn: 348568
2018-12-07 06:23:06 +00:00
Dean Michael Berris
190c49bc8f Re-land "[XRay] Move-only Allocator, FunctionCallTrie, and Array"
This reverts commit r348455, with some additional changes:

- Work-around deficiency of gcc-4.8 by duplicating the implementation of
  `AppendEmplace` in `Append`, but instead of using brace-init for the
  copy construction, use a placement new explicitly calling the copy
  constructor.

llvm-svn: 348563
2018-12-07 03:19:13 +00:00
Dean Michael Berris
82f7b21f17 Revert "[XRay] Move-only Allocator, FunctionCallTrie, and Array"
This reverts commits r348438, r348445, and r348449 due to breakages with
gcc-4.8 builds.

llvm-svn: 348455
2018-12-06 03:28:57 +00:00
Dean Michael Berris
cb447a2604 Re-land r348335 "[XRay] Move-only Allocator, FunctionCallTrie, and Array"
Continuation of D54989.

Additional changes:

  - Use `.AppendEmplace(...)` instead of `.Append(Type{...})` to appease
    GCC 4.8 with confusion on when an initializer_list is used as
    opposed to a temporary aggregate initialized object.

llvm-svn: 348438
2018-12-06 00:25:56 +00:00
Hans Wennborg
83ff22c297 Revert r348335 "[XRay] Move-only Allocator, FunctionCallTrie, and Array"
.. and also the follow-ups r348336 r348338.

It broke stand-alone compiler-rt builds with GCC 4.8:

In file included from /work/llvm/projects/compiler-rt/lib/xray/xray_function_call_trie.h:20:0,
                 from /work/llvm/projects/compiler-rt/lib/xray/xray_profile_collector.h:21,
                 from /work/llvm/projects/compiler-rt/lib/xray/xray_profile_collector.cc:15:
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h: In instantiation of ‘T* __xray::Array<T>::AppendEmplace(Args&& ...) [with Args = {const __xray::FunctionCallTrie::mergeInto(__xray::FunctionCallTrie&) const::NodeAndTarget&}; T = __xray::FunctionCallTrie::mergeInto(__xray::FunctionCallTrie&) const::NodeAndTarget]’:
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:383:71:   required from ‘T* __xray::Array<T>::Append(const T&) [with T = __xray::FunctionCallTrie::mergeInto(__xray::FunctionCallTrie&) const::NodeAndTarget]’
/work/llvm/projects/compiler-rt/lib/xray/xray_function_call_trie.h:517:54:   required from here
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:378:5: error: could not convert ‘{std::forward<const __xray::FunctionCallTrie::mergeInto(__xray::FunctionCallTrie&) const::NodeAndTarget&>((* & args#0))}’ from ‘<brace-enclosed initializer list>’ to ‘__xray::FunctionCallTrie::mergeInto(__xray::FunctionCallTrie&) const::NodeAndTarget’
     new (AlignedOffset) T{std::forward<Args>(args)...};
     ^
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h: In instantiation of ‘T* __xray::Array<T>::AppendEmplace(Args&& ...) [with Args = {const __xray::profileCollectorService::{anonymous}::ThreadTrie&}; T = __xray::profileCollectorService::{anonymous}::ThreadTrie]’:
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:383:71:   required from ‘T* __xray::Array<T>::Append(const T&) [with T = __xray::profileCollectorService::{anonymous}::ThreadTrie]’
/work/llvm/projects/compiler-rt/lib/xray/xray_profile_collector.cc:98:34:   required from here
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:378:5: error: could not convert ‘{std::forward<const __xray::profileCollectorService::{anonymous}::ThreadTrie&>((* & args#0))}’ from
‘<brace-enclosed initializer list>’ to ‘__xray::profileCollectorService::{anonymous}::ThreadTrie’
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h: In instantiation of ‘T* __xray::Array<T>::AppendEmplace(Args&& ...) [with Args = {const __xray::profileCollectorService::{anonymous}::ProfileBuffer&}; T = __xray::profileCollectorService::{anonymous}::ProfileBuffer]’:
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:383:71:   required from ‘T* __xray::Array<T>::Append(const T&) [with T = __xray::profileCollectorService::{anonymous}::ProfileBuffer]
’
/work/llvm/projects/compiler-rt/lib/xray/xray_profile_collector.cc:244:44:   required from here
/work/llvm/projects/compiler-rt/lib/xray/xray_segmented_array.h:378:5: error: could not convert ‘{std::forward<const __xray::profileCollectorService::{anonymous}::ProfileBuffer&>((* & args#0))}’ from ‘<brace-enclosed initializer list>’ to ‘__xray::profileCollectorService::{anonymous}::ProfileBuffer’

> Summary:
> This change makes the allocator and function call trie implementations
> move-aware and remove the FunctionCallTrie's reliance on a
> heap-allocated set of allocators.
>
> The change makes it possible to always have storage associated with
> Allocator instances, not necessarily having heap-allocated memory
> obtainable from these allocator instances. We also use thread-local
> uninitialised storage.
>
> We've also re-worked the segmented array implementation to have more
> precondition and post-condition checks when built in debug mode. This
> enables us to better implement some of the operations with surrounding
> documentation as well. The `trim` algorithm now has more documentation
> on the implementation, reducing the requirement to handle special
> conditions, and being more rigorous on the computations involved.
>
> In this change we also introduce an initialisation guard, through which
> we prevent an initialisation operation from racing with a cleanup
> operation.
>
> We also ensure that the ThreadTries array is not destroyed while copies
> into the elements are still being performed by other threads submitting
> profiles.
>
> Note that this change still has an issue with accessing thread-local
> storage from signal handlers that are instrumented with XRay. We also
> learn that with the testing of this patch, that there will be cases
> where calls to mmap(...) (through internal_mmap(...)) might be called in
> signal handlers, but are not async-signal-safe. Subsequent patches will
> address this, by re-using the `BufferQueue` type used in the FDR mode
> implementation for pre-allocated memory segments per active, tracing
> thread.
>
> We still want to land this change despite the known issues, with fixes
> forthcoming.
>
> Reviewers: mboerger, jfb
>
> Subscribers: jfb, llvm-commits
>
> Differential Revision: https://reviews.llvm.org/D54989

llvm-svn: 348346
2018-12-05 10:19:55 +00:00
Dean Michael Berris
adc880467d [XRay] Move-only Allocator, FunctionCallTrie, and Array
Summary:
This change makes the allocator and function call trie implementations
move-aware and remove the FunctionCallTrie's reliance on a
heap-allocated set of allocators.

The change makes it possible to always have storage associated with
Allocator instances, not necessarily having heap-allocated memory
obtainable from these allocator instances. We also use thread-local
uninitialised storage.

We've also re-worked the segmented array implementation to have more
precondition and post-condition checks when built in debug mode. This
enables us to better implement some of the operations with surrounding
documentation as well. The `trim` algorithm now has more documentation
on the implementation, reducing the requirement to handle special
conditions, and being more rigorous on the computations involved.

In this change we also introduce an initialisation guard, through which
we prevent an initialisation operation from racing with a cleanup
operation.

We also ensure that the ThreadTries array is not destroyed while copies
into the elements are still being performed by other threads submitting
profiles.

Note that this change still has an issue with accessing thread-local
storage from signal handlers that are instrumented with XRay. We also
learn that with the testing of this patch, that there will be cases
where calls to mmap(...) (through internal_mmap(...)) might be called in
signal handlers, but are not async-signal-safe. Subsequent patches will
address this, by re-using the `BufferQueue` type used in the FDR mode
implementation for pre-allocated memory segments per active, tracing
thread.

We still want to land this change despite the known issues, with fixes
forthcoming.

Reviewers: mboerger, jfb

Subscribers: jfb, llvm-commits

Differential Revision: https://reviews.llvm.org/D54989

llvm-svn: 348335
2018-12-05 06:44:34 +00:00
Dean Michael Berris
388af45f18 [XRay] Add a test for allocator exhaustion
Use a more representative test of allocating small chunks for
oddly-sized (small) objects from an allocator that has a page's worth of
memory.

llvm-svn: 347286
2018-11-20 03:56:04 +00:00
Dean Michael Berris
ba02cb58cf [XRay] Move buffer extents back to the heap
Summary:
This change addresses an issue which shows up with the synchronised race
between threads writing into a buffer, and another thread reading the
buffer.

In a lot of cases, we cannot guarantee that threads will always see the
signal to finalise their buffers in time despite the grace periods and
state machine maintained through atomic variables. This change addresses
it by ensuring that the same instance being updated to indicate how much
of the buffer is "used" by the writing thread is the same instance being
read by the thread processing the buffer to be written out to disk or
handled through the iterators.

To do this, we ensure that all the "extents" instances live in their own
the backing store, in a different contiguous page from the
buffer-specific backing store. We also take precautions to ensure that
the atomic variables are cache-line-sized to prevent false-sharing from
unnecessarily causing cache contention on unrelated writes/reads.

It's feasible that we may in the future be able to move the storage of
the extents objects into the single backing store, slightly changing the
way to compute the size(s) of the buffers, but in the meantime we'll
settle for the isolation afforded by having a different backing store
for the extents instances.

Reviewers: mboerger

Subscribers: jfb, llvm-commits

Differential Revision: https://reviews.llvm.org/D54684

llvm-svn: 347280
2018-11-20 01:00:26 +00:00
Dean Michael Berris
9e089fca7e [XRay] Add a test for function id encoding/decoding (NFC)
Increase test coverage for function enter/exit encoding/decoding.

llvm-svn: 346477
2018-11-09 07:43:30 +00:00
Dean Michael Berris
e39a89fbfb [XRay] Fix enter function tracing for record unwriting
Summary:
Before this change, we could run into a situation where we may try to
undo tail exit records after writing metadata records before a function
enter event. This change rectifies that by resetting the tail exit
counter after writing the metadata records.

Reviewers: mboerger

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D54292

llvm-svn: 346475
2018-11-09 06:49:00 +00:00
Dean Michael Berris
da375a67f8 [XRay] Improve FDR trace handling and error messaging
Summary:
This change covers a number of things spanning LLVM and compiler-rt,
which are related in a non-trivial way.

In LLVM, we have a library that handles the FDR mode even log loading,
which uses C++'s runtime polymorphism feature to better faithfully
represent the events that are written down by the FDR mode runtime. We
do this by interpreting a trace that's serliased in a common format
agreed upon by both the trace loading library and the FDR mode runtime.
This library is under active development, which consists of features
allowing us to reconstitute a higher-level event log.

This event log is used by the conversion and visualisation tools we have
for interpreting XRay traces.

One of the tools we have is a diagnostic tool in llvm-xray called
`fdr-dump` which we've been using to debug our expectations of what the
FDR runtime should be writing and what the logical FDR event log
structures are. We use this fairly extensively to reason about why some
non-trivial traces we're generating with FDR mode runtimes fail to
convert or fail to parse correctly.

One of these failures we've found in manual debugging of some of the
traces we've seen involve an inconsistency between the buffer extents (a
record indicating how many bytes to follow are part of a logical
thread's event log) and the record of the bytes written into the log --
sometimes it turns out the data could be garbage, due to buffers being
recycled, but sometimes we're seeing the buffer extent indicating a log
is "shorter" than the actual records associated with the buffer. This
case happens particularly with function entry records with a call
argument.

This change for now updates the FDR mode runtime to write the bytes for
the function call and arg record before updating the buffer extents
atomically, allowing multiple threads to see a consistent view of the
data in the buffer using the atomic counter associated with a buffer.
What we're trying to prevent here is partial updates where we see the
intermediary updates to the buffer extents (function record size then
call argument record size) becoming observable from another thread, for
instance, one doing the serialization/flushing.

To do both diagnose this issue properly, we need to be able to honour
the extents being set in the `BufferExtents` records marking the
beginning of the logical buffers when reading an FDR trace. Since LLVM
doesn't use C++'s RTTI mechanism, we instead follow the advice in the
documentation for LLVM Style RTTI
(https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on
this RTTI feature to ensure that our file-based record producer (our
streaming "deserializer") can honour the extents of individual buffers
as we interpret traces.

This also sets us up to be able to eventually do smart
skipping/continuation of FDR logs, seeking instead to find BufferExtents
records in cases where we find potentially recoverable errors. In the
meantime, we make this change to operate in a strict mode when reading
logical buffers with extent records.

Reviewers: mboerger

Subscribers: hiraditya, llvm-commits, jfb

Differential Revision: https://reviews.llvm.org/D54201

llvm-svn: 346473
2018-11-09 06:26:48 +00:00
Dean Michael Berris
25f8d204b8 [XRay] Update XRayRecord to support Custom/Typed Events
Summary:
This change cuts across LLVM and compiler-rt to add support for
rendering custom events in the XRayRecord type, to allow for including
user-provided annotations in the output YAML (as raw bytes).

This work enables us to add custom event and typed event records into
the `llvm::xray::Trace` type for user-provided events. This can then be
programmatically handled through the C++ API and can be included in some
of the tooling as well. For now we support printing the raw data we
encounter in the custom events in the converted output.

Future work will allow us to start interpreting these custom and typed
events through a yet-to-be-defined API for extending the trace analysis
library.

Reviewers: mboerger

Subscribers: hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D54139

llvm-svn: 346214
2018-11-06 08:51:37 +00:00
Dean Michael Berris
1e255e7a7b [XRay] Update TSC math to handle wraparound
Summary:
Prior to this change, we can run into situations where the TSC we're
getting when exiting a function is less than the TSC we got when
entering it. This would sometimes cause the counter for cumulative call
times overflow, which was erroneously also being stored as a signed
64-bit integer.

This change addresses both these issues while adding provisions for
tracking CPU migrations. We do this because moving from one CPU to
another doesn't guarantee that the timestamp counter for some
architectures aren't guaranteed to be synchronised. For the moment, we
leave the provisions there until we can update the data format to
include the counting of CPU migrations we can catch.

We update the necessary tests as well, ensuring that our expectations
for the cycle accounting to be met in case of counter wraparound.

Reviewers: mboerger

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D54088

llvm-svn: 346116
2018-11-05 05:43:22 +00:00
Dean Michael Berris
0c96ae3d6e [XRay] Update delta computations in runtime
Summary:
Fix some issues discovered from mostly manual inspection of outputs from
the `llvm-xray fdr-dump` tool.

It turns out we haven't been writing the deltas properly, and have been
writing down zeros for deltas of some records. This change fixes this
oversight born by the recent refactoring.

Reviewers: mboerger

Subscribers: llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D54022

llvm-svn: 345954
2018-11-02 08:07:38 +00:00
Dean Michael Berris
e8c650ab12 [XRay] Fix TSC and atomic custom/typed event accounting
Summary:
This is a follow-on change to D53858 which turns out to have had a TSC
accounting bug when writing out function exit records in FDR mode.

This change adds a number of tests to ensure that:

- We are handling the delta between the exit TSC and the last TSC we've
  seen.

- We are writing the custom event and typed event records as a single
  update to the buffer extents.

- We are able to catch boundary conditions when loading FDR logs.

We introduce a TSC matcher to the test helpers, which we use in the
testing/verification of the TSC accounting change.

Reviewers: mboerger

Subscribers: mgorny, hiraditya, jfb, llvm-commits

Differential Revision: https://reviews.llvm.org/D53967

llvm-svn: 345905
2018-11-01 22:57:50 +00:00
Dean Michael Berris
36d9746630 [XRay] Use more portable control block
Summary:
In D53560, we assumed a specific layout for memory without using an
explicit structure. This follow-up change uses more portable layout
control by using unions in a struct, and consolidating the memory
management code in the buffer queue.

We also take the opportunity to improve the documentation on the types
and operations, along with simplifying some of the logic in the buffer
queue implementation.

Reviewers: mboerger, eizan

Subscribers: jfb, llvm-commits

Differential Revision: https://reviews.llvm.org/D53802

llvm-svn: 345485
2018-10-29 02:18:14 +00:00
Dean Michael Berris
18024786b9 [XRay] Support generational buffers in FDR controller
Summary:
This is an intermediary step in the full support for generational buffer
management in the FDR runtime. This change makes the FDR controller
aware of the new generation number in the buffers handed out by the
BufferQueue type.

In the process of making this change, we've realised that the cleanest
way of ensuring that the backing store per generation is live while all
the threads that need access to it will need reference counting to tie
the backing store to the lifetime of all threads that have a handle on
buffers associated with the memory.

We also learn that we're missing the edge-case in the function exit
handler's implementation where the first record being written into the
buffer is a function exit, which is caught/fixed by the test for
generational buffer management.

We still haven't wired the controller into the FDR mode runtime, which
will need the reference counting on the backing store implemented to
ensure that we're being conservatively thread-safe with this approach.

Depends on D52974.

Reviewers: mboerger, eizan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D53551

llvm-svn: 345445
2018-10-27 03:00:21 +00:00
Dean Michael Berris
788b17ca78 [XRay][compiler-rt] Generational Buffer Management
Summary:
This change updates the buffer queue implementation to support using a
generation number to identify the lifetime of buffers. This first part
introduces the notion of the generation number, without changing the way
we handle the buffers yet.

What's missing here is the cleanup of the buffers. Ideally we'll keep
the two most recent generations. We need to ensure that before we do any
writes to the buffers, that we check the generation number(s) first.

Those changes will follow-on from this change.

Depends on D52588.

Reviewers: mboerger, eizan

Subscribers: llvm-commits, jfb

Differential Revision: https://reviews.llvm.org/D52974

llvm-svn: 344881
2018-10-22 04:53:58 +00:00
Dean Michael Berris
ebfbf89000 [XRay] Handle allocator exhaustion in segmented array
Summary:
This change allows us to handle allocator exhaustion properly in the
segmented array implementation. Before this change, we relied on the
caller of the `trim` function to provide a valid number of elements to
trim. This change allows us to do the right thing in case the elements
to trim is greater than the size of the container.

Reviewers: mboerger, eizan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D53484

llvm-svn: 344880
2018-10-22 02:11:27 +00:00