As per PEP-0394[1], there is no real concensus over what binary names
Python has, specifically 'python' could be Python 3, Python 2, or not
exist.
However, everyone has a python3 interpreter and the scripts are all
written for Python 3. Unify the shebangs so that the ~50% of shebangs
that use python now use python3.
[1] https://peps.python.org/pep-0394/
For a full toolchain supporting execute-only code generation the runtime
libraries also need to be pre-compiled with it enabled. The generic
`RUNTIMES_EXECUTE_ONLY_CODE` CMake option can now be used during build
configuration to enable execute-only code generation in compiler-rt.
The build option can only be enabled for a runtimes build of
compiler-rt, because a recent version of Clang is needed to correctly
compile assembly files with execute-only code support.
Related RFC:
https://discourse.llvm.org/t/rfc-execute-only-code-support-for-runtime-libraries-on-aarch64/86180
The following assertion was being triggered:
```
assert.h assertion failed at llvm-project/compiler-rt/lib/fuzzer/FuzzerTracePC.cpp:237 in void fuzzer::TracePC::UpdateObservedPCs(): M.Size() == (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start)
```
# The bug
When built with `-fsanitize=fuzzer`, each “module” (.so file, or the
binary itself) will be instrumented, and when loaded into the process
will make a call to these two functions:
- `__sanitizer_cov_8bit_counters_init`
- `__sanitizer_cov_pcs_init`
Each of these is called with start and end pointers defining an array.
In libFuzzer, these functions are implemented with
`HandleInline8bitCountersInit` and `HandlePCsInit`. Each of them pushes
back the provided pointers into a separate array, `Modules` and
`ModulePCTable` respectively. These arrays are meant to be kept in-sync;
index i into Modules should refer to the same `.so` as index i into
ModulePCTable. The assertion was triggering because these lists got
out-of-sync.
The problem is that the 8bit handler contains this line:
```
if (Start == Stop) return;
```
but the PC handler contains no such corresponding line. This meant that
if a module was ever instrumented but “empty” (its 8bit counter and PC
arrays were both of length 0), then its PC array would still be added
but its 8bit counter array would not.
# Why this issue was never seen before
The circumstances to trigger this issue are unusual:
- You need a compilation unit that doesn't contain any code (though it
may contain global variable declarations and similar). That doesn't
happen very often.
- That compilation unit must be dynamically linked, not statically
linked. If statically linked, it’ll be merged into a single “module”
with the main binary, and the arrays will be merged as well; you won’t
end up with length-0 arrays.
- To notice the issue, assertions must be enabled. If disabled,
libFuzzer will be buggy (it may have worse coverage), but it won't
crash, and "worse coverage" is extremely unlikely to be noticed.
# This change
This change solves the issue by adding the same `if (Start == Stop)
return;` check to `HandlePCsInit`. This prevents the arrays from getting
out-of-sync. This change also adds a test that identifies the previous
issue when compiled with assertions enabled, but now passes with the
fix.
Every once in a couple hundred runs of a downstream fuzzer test, we see
a fuzzing test freeze while waiting for a thread to be suspended. The
main thread is frozen because it's waiting to suspend either the alarm
or rss thread which is stuck waiting for an exception they sent out to
be handled. Specifically, both threads send out a synthetic
`ZX_EXCP_THREAD_STARTING` exception to be handled by the crash handling
thread which sets up an exception channel on the whole process with
`ZX_EXCEPTION_CHANNEL_DEBUGGER`. This is the only channel type that
listens to thread stop/start exceptions. Normally, the exception would
be ignored and the alarm or rss thread would continue normally once the
crash handling thread closes the read exception. However, the memory
snapshot machinery can suspend this thread while its in the process of
waiting for or handling a `ZX_EXCP_THREAD_STARTING` sent by either the
rss or alarm thread. If this is suspended first, then we attempt to
suspend either the alarm or rss thread while they're still waiting for
the crash handling thread to handle its exception, we will freeze
waiting for those threads to give the suspend signal, which they won't
because they're blocked on waiting for the exception handler. This is
the deadlock.
Until there's a way for the memory snapshot machinery to suspend the
thread while it's stuck on an exception, then we can work around this in
the meantime by just ensuring the alarm and rss threads start normally
via signals on the initial startup path. I can assert locally the
freezing doesn't occur after 6000 runs where prior we would see it every
couple hundred runs. Note this type of issue can arise again if the
fuzzing test launches any dangling threads that happen to not start yet.
One of the recommendations for writing a fuzz test is that the test may
launch threads, but they should be joined by the end of the test
(https://llvm.org/docs/LibFuzzer.html#fuzz-target), so hopefully we
won't see this type of bug rise frequently from fuzz tests. More
broadly, this can also arise if any process launches its own debugger
via `ZX_EXCEPTION_CHANNEL_DEBUGGER`, but I would think in practice this
isn't very likely to happen.
More context in https://fxbug.dev/436923423.
---------
Co-authored-by: Petr Hosek <phosek@google.com>
Reverts llvm/llvm-project#147422
Seems to be causing problems with tracebacks. Probably the trackback
code doesn't know how to switch back to the regular stack after it gets
to the top of the signal stack.
SA_ONSTACK is required for certain runtimes that use small stacks, for
instance the Go runtime.
See https://github.com/golang/go/issues/49075
SA_ONSTACK is a no-op unless someone also calls sigaltstack.
Fix for #144495 by 6f4add3 broke sanitizer-aarch64-linux buildbot.
compiler-rt/lib/fuzzer/tests build failed because the linker was
looking gcc_s without '-l' appended.
The CMake script was adding the library name without the required
'-l' prefix. This patch adds the -l prefix changing gcc_s to -lgcc_s
and gcc to -lgcc.
https://lab.llvm.org/buildbot/#/builders/51/builds/18170
compiler-rt/lib/fuzzer/tests build was failing on armv7, with undefined
references to unwinder symbols, such as __aeabi_unwind_cpp_pr0.
This occurs because the test is built with `-nostdlib++` but `libunwind`
is not explicitly linked to the final test executable.
This patch resolves the issue by adding CMake logic to explicitly link
the required unwinder to the fuzzer tests, inspired by the same solution
used to fix Scudo build failures by https://reviews.llvm.org/D142888.
The INSTALL_BYPRODUCTS ExternalProject_Add() argument was only added in
CMake 3.26 and the current minimum is 3.20. Work around this by using an
explicit ExternalProject_Add_Step() call for the install step with a
BYPRODUCTS argument. We can't keep using the `install` name here since that
is reserved by the CMake implementation and results in errors when used.
This commit should be reverted once LLVM depends on CMake 3.26.
Pull Request: https://github.com/llvm/llvm-project/pull/115677
Since b4130bee6bfd34d8045f02fc9f951bcb5db9d85c, we check for
_LIBCPP_HAS_THREAD_API_PTHREAD to decide between using
SetThreadDescription or pthread_setname_np for setting the thread name.
c6f3b7bcd0596d30f8dabecdfb9e44f9a07b6e4c changed how libcxx defines
their configuration macros - now they are always defined, but defined to
0 or 1, while they previously were either defined or undefined.
As these libcxx defines used to be defined to an empty string (rather
than expanding to 1) if enabled, we can't easily produce an expression
that works both with older and newer libcxx. Additionally, these defines
are libcxx internal config macros that aren't a detail that isn't
supported and isn't meant to be relied upon.
Simply skip trying to set thread name on MinGW as we can't easily know
which kind of thread native handle we have. Setting the thread name is
only a nice to have, quality of life improvement - things should work
the same even without it.
Additionally, libfuzzer isn't generally usable on MinGW targets yet
(Clang doesn't include it in the getSupportedSanitizers() method for the
MinGW target), so this shouldn't make any difference in practice anyway.
Using the build tree is somewhat fragile since the layout is not
guaranteed to be stable and means the tests are tightly coupled to the
libc++/libc++abi build tree layout. Instead update the ExternalProject
to install the library and headers and do not add the build tree to
the include/linker flags.
Pull Request: https://github.com/llvm/llvm-project/pull/115077
The `check-fuzzer` runs fine with cl build llvm, but the following lit
tests fail with clang-cl build llvm
```
********************
Timed Out Tests (2):
libFuzzer-x86_64-default-Windows :: fork-ubsan.test
libFuzzer-x86_64-default-Windows :: fuzzer-oom.test
********************
Failed Tests (22):
libFuzzer-x86_64-default-Windows :: acquire-crash-state.test
libFuzzer-x86_64-default-Windows :: cross_over_copy.test
libFuzzer-x86_64-default-Windows :: cross_over_insert.test
libFuzzer-x86_64-default-Windows :: exit_on_src_pos.test
libFuzzer-x86_64-default-Windows :: fuzzer-alignment-assumption.test
libFuzzer-x86_64-default-Windows :: fuzzer-implicit-integer-sign-change.test
libFuzzer-x86_64-default-Windows :: fuzzer-implicit-signed-integer-truncation-or-sign-change.test
libFuzzer-x86_64-default-Windows :: fuzzer-implicit-signed-integer-truncation.test
libFuzzer-x86_64-default-Windows :: fuzzer-implicit-unsigned-integer-truncation.test
libFuzzer-x86_64-default-Windows :: fuzzer-printcovpcs.test
libFuzzer-x86_64-default-Windows :: fuzzer-timeout.test
libFuzzer-x86_64-default-Windows :: fuzzer-ubsan.test
libFuzzer-x86_64-default-Windows :: minimize_crash.test
libFuzzer-x86_64-default-Windows :: minimize_two_crashes.test
libFuzzer-x86_64-default-Windows :: null-deref-on-empty.test
libFuzzer-x86_64-default-Windows :: null-deref.test
libFuzzer-x86_64-default-Windows :: print-func.test
libFuzzer-x86_64-default-Windows :: stack-overflow-with-asan.test
libFuzzer-x86_64-default-Windows :: trace-malloc-2.test
libFuzzer-x86_64-default-Windows :: trace-malloc-unbalanced.test
libFuzzer-x86_64-default-Windows :: trace-malloc.test
```
The related commits are
53a81d4d26
and
e31efd8f6f.
Following the change in
e31efd8f6f
can fix these failures.
As for the issue mentioned in the comment that alternatename support in
clang not good enough(https://bugs.llvm.org/show_bug.cgi?id=40218). I
find that using `__builtin_function_start(func)` instead of directly
using `func` would make it work as intended.
In C, it is UB to call `memcmp(NULL, NULL, 0)`, `memcpy(NULL, NULL, 0)`,
etc. Unfortunately, `(NULL, 0)` is the natural representation of an
empty sequence of objects and extremely common in real world code. As a
result, all C code, and C++ code which calls into C functions, must
carefully guard all calls to `memcpy`.
This is a serious, real world usability issue in C and should be fixed
in the language (see #49459). In the meantime, pay the cost of the extra
branch to avoid tripping UBSan in libFuzzer. Once the usability problem
in C has been fixed, these checks can be removed.
Fixes#96772
This fixes a few of these warnings, when building with Clang ToT on
Windows:
```
[622/7618] Building CXX object
projects\compiler-rt\lib\sanitizer_common\CMakeFiles\RTSanitizerCommonSymbolizer.x86_64.dir\sanitizer_symbolizer_win.cpp.obj
C:\src\git\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_symbolizer_win.cpp(74,3):
warning: cast from 'FARPROC' (aka 'long long (*)()') to
'decltype(::StackWalk64) *' (aka 'int (*)(unsigned long, void *, void *,
_tagSTACKFRAME64 *, void *, int (*)(void *, unsigned long long, void *,
unsigned long, unsigned long *), void *(*)(void *, unsigned long long),
unsigned long long (*)(void *, unsigned long long), unsigned long long
(*)(void *, void *, _tagADDRESS64 *))') converts to incompatible
function type [-Wcast-function-type-mismatch]
```
This is similar to https://github.com/llvm/llvm-project/pull/97905
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
libfuzzer's -jobs option will, depending on the number of CPUs, spin up
a
WorkerThread and end up printing the log file using CopyFileToErr.
This leads to an MSan false positive. This patch disables the MSan
interceptor checks,
similarly to other instances in https://reviews.llvm.org/D48891
Side-note: this false positive issue first appeared when printf()
was replaced by puts() (90b4d1bcb20180c591385131b12fa90d2e4860b1).
The interceptor check was always present; however, MSan does not
check_printf by default.
All these unit tests already include ${COMPILER_RT_GTEST_SOURCE} as an
input source file and the target llvm_gtest does not exist for
standalone builds. Currently the DEPS argument is ignored for standalone
builds so the missing target is not a problem, but as part of fixing a
build race for standalone builds I am planning to include those
dependencies in COMPILER_RT_TEST_STANDALONE_BUILD_LIBS configurations.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83649
SetThreadDescription symbol needs to be dynamically loaded before usage.
Then using a wide string buffer, since we re using a null terminated
string, we can use MultiByteToWideChar -1 as 4th argument to finally set
the thread name.
Previously `SetThreadDescription` was called directly causing crash.
It was reverted in dd3aa26fc8e9de37a39611f7a6a602bcb4153784
