Currently, the testsuite uses the default runtimes path to find the
runtimes libraries which may or may not match the just-built runtimes.
This change uses the `-resource-dir` flag for clang whenever
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` is set to ensure that we are
actually testing the currently built libraries rather than the ones
bundled with `${COMPILER_RT_TEST_COMPILER}`.
The existing logic works fine when clang and compiler-rt share the same
build directory ``-DLLVM_ENABLE_PROJECTS=clang;compiler-rt`, but when
building compiler-rt separately we need to tell the compiler used for
the tests where it can find the just-built libraries.
This reduces the fixes check-all failures to one in my configuration:
```
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -G Ninja
-DCMAKE_C_COMPILER=$HOME/output/upstream-llvm/bin/clang
-DCMAKE_CXX_COMPILER=$HOME/output/upstream-llvm/bin/clang++
-DCOMPILER_RT_INCLUDE_TESTS=ON
-DLLVM_EXTERNAL_LIT=$HOME/build/upstream-llvm-project-build/bin/llvm-lit
-DLLVM_CMAKE_DIR=$HOME/output/upstream-llvm
-DCOMPILER_RT_DEBUG=OFF
-S $HOME/src/upstream-llvm-project/compiler-rt
-B $HOME/src/upstream-llvm-project/compiler-rt/cmake-build-all-sanitizers
```
Reviewed By: vitalybuka, delcypher, MaskRay
Pull Request: https://github.com/llvm/llvm-project/pull/83088
The HOST_LINK_VERSION is a hardcoded string in Darwin clang that detects
the linker version at configure time. The driver uses this information
to build the correct set of arguments for the linker. This patch detects
the linker version again during compiler-rt configuration and passes it
to the libfuzzer tests. This allows a clang built on a machine with a
new linker to run compiler-rt tests on a machine with an old linker.
rdar://125932376
With https://github.com/llvm/llvm-project/pull/83088, we now need the
runtimes to be built before running test if
COMPILER_RT_TEST_STANDALONE_BUILD_LIBS is true, since otherwise we
get failures running `ninja check-all` such as the following:
```
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.fuzzer-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-basic-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-fdr-x86_64.a: No such file or directory
```
This is a follow-up to 058e9b03 which started removing these checks
and it should make it easier to stop forcing COMPILER_RT_STANDALONE_BUILD
for runtimes builds in the future.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83651
The FileCheck string `LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS:
{{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}` is too restrictive and may
fail the test in some case.
If we look at the commit that added this
check(66df98945e),
This check is for printing out the long mutation sequence, such as this
one
```
#53552 REDUCE cov: 6 ft: 6 corp: 5/9b lim: 4096 exec/s: 0 rss: 37Mb L: 2/3 MS: 54 ChangeByte-PersAutoDict-ChangeBit-ChangeBinInt-ChangeBit-ChangeBit-ChangeByte-CMP-EraseBytes-EraseBytes-CrossOver-InsertRepeatedBytes-ChangeByte-EraseBytes-InsertRepeatedBytes-ShuffleBytes-ChangeByte-ShuffleBytes-ChangeBit-CrossOver-ChangeBit-ShuffleBytes-ChangeBinInt-ShuffleBytes-EraseBytes-InsertByte-Custom-ShuffleBytes-CopyPart-InsertRepeatedBytes-PersAutoDict-InsertRepeatedBytes-ChangeByte-CrossOver-CrossOver-PersAutoDict-PersAutoDict-EraseBytes-ChangeBit-CopyPart-ChangeByte-CopyPart-InsertRepeatedBytes-CrossOver-CrossOver-CrossOver-CrossOver-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-EraseBytes-Custom- DE: "\377\377"-"\001\000"-"\001\000"-"\000\000\000\000\000\000\000\000"-"\001\000\000\000"-
```
But if we look at the code doing the printing
```cpp
void MutationDispatcher::PrintMutationSequence(bool Verbose) {
Printf("MS: %zd ", CurrentMutatorSequence.size());
size_t EntriesToPrint =
Verbose ? CurrentMutatorSequence.size()
: std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
for (size_t i = 0; i < EntriesToPrint; i++)
Printf("%s-", CurrentMutatorSequence[i].Name);
if (!CurrentDictionaryEntrySequence.empty()) {
Printf(" DE: ");
EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
: std::min(kMaxMutationsToPrint,
CurrentDictionaryEntrySequence.size());
for (size_t i = 0; i < EntriesToPrint; i++) {
Printf("\"");
PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
}
}
}
```
We can see that the `DE: XXX` is not always printed. So the following
output is possible(and is from real-life failure), notince the missing
of `DE: XXX`.
```
#13613 NEW cov: 5 ft: 5 corp: 4/6b lim: 4096 exec/s: 0 rss: 32Mb L: 2/2 MS: 27 InsertByte-ChangeBinInt-ChangeBinInt-CrossOver-ShuffleBytes-ChangeBit-EraseBytes-ShuffleBytes-InsertByte-InsertRepeatedBytes-CopyPart-InsertByte-ChangeByte-ChangeBit-InsertByte-CrossOver-EraseBytes-CopyPart-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-CrossOver-ShuffleBytes-ChangeBit-Custom-
#13765 ......
```
This output is totally legit and will fail that check.
So I remove the check for the following strings, I think `MS: {{[0-9]*}}
{{(([a-zA-Z]*-){11,})}}` is sufficient for checking the long mutation
sequence. This should help resolve the flaky failure of
fuzzer-custommutator.test.
Add config name for fuzzer lit test, to make it easier to identify failures are with which config.
Before this change, same lit tests with different configs will share the same test name.
```
********************
Failed Tests (2):
libFuzzer :: fuzzer-flags.test
libFuzzer :: fuzzer-flags.test
```
Actually this is a failure of two lit tests(two configs of the same test).
With this change, the names will be different.
```
********************
Failed Tests (2):
libFuzzer-i386-default-Linux ::fuzzer-flags.test
libFuzzer-x86_64-default-Linux :: fuzzer-flags.test
```
Reviewed By: MaskRay, vitalybuka
Differential Revision: https://reviews.llvm.org/D158696
The test requires that an integer division by zero causes a trap, leading to a
signal like SIGFPE. This is the case on x86. On many other architectures, such
as Arm (unless software division is used and `__aeabi_idiv0` is patched),
Power, RISC-V, LoongArch, there is no trap. Therefore, it's more appropriate to
invert the condition to run the test.
Reviewed By: Ami-zhang
Differential Revision: https://reviews.llvm.org/D155644
This is an ongoing series of commits that are reformatting our
Python code. This catches the last of the python files to
reformat. Since they where so few I bunched them together.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: jhenderson, #libc, Mordante, sivachandra
Differential Revision: https://reviews.llvm.org/D150784
This test is heavy on test resources and involves moving a large 2GB+ file
across an ssh connection when testing on remote devices. Exclude all tests
except for macOS testing (on host). Remote device testing for macOS may
eventually cause a problem on macOS as well w/o better handling of tests
that require more resource/time than we currently handle.
Introduced by:
https://reviews.llvm.org/D146189
Subsequent exclusion of all but darwin by:
https://reviews.llvm.org/D147094
Subsequent exclusion of tvOS and watchOS by:
https://reviews.llvm.org/D147502
Subsequent exclusion of all but macOS...
rdar://107570309
Reviewed By: yln, thetruestblue
Differential Revision: https://reviews.llvm.org/D148727
The test "big-file-copy.test" introduced in D146189 and constrained to darwin by
D147094, is by this differential further constrained to only those devices with
sufficient resources. Also correct the test to read the environment variable
"result" from the same shell in which it was stored (which may differ on devices).
Reviewed By: thetruestblue
Differential Revision: https://reviews.llvm.org/D147502
This test has to be limited to darwin due to multiple failures on other
platforms for multple reasons. (Timeout, puts() limit, etc.). This
commit modifies D146189.
Reviewed By: NoQ
Differential Revision: https://reviews.llvm.org/D147094
CopyFileToErr() uses Printf("%s", ...) which fails with a negative size on
files >2Gb (Its path is through var-args wrappers to an unnecessary "%s"
expansion and subject to int overflows) Using puts() in place of printf()
bypasses this path and writes the string directly to stderr. This avoids the
present loss of data when a crashed worker has generated >2Gb of output.
rdar://99384640
Reviewed By: yln, rsundahl
Differential Revision: https://reviews.llvm.org/D146189
There is test for ft file-id in control file, but no test for cov line.
Without the test, a invalid cov file-id would cause crash.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D145672
CopyFileToErr() uses Printf("%s", ...) which fails with a negative size on
files >2Gb (Its path is through var-args wrappers to an unnecessary "%s"
expansion and subject to int overflows) Using puts() in place of printf()
bypasses this path and writes the string directly to stderr. This avoids the
present loss of data when a crashed worker has generated >2Gb of output.
rdar://99384640
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D146189
`CollectDataFlow()` uses `Flags.collect_data_flow` and
`Flags.data_flow_trace` at the same time. But in the null check before
the invocation, only `Flags.collect_data_flow` is checked, and there is
no other method to make sure `Flags.data_flow_trace` is not null, so
adding a null check for `Flags.data_flow_trace`.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D145040
These four tests are failing on tvOS devices (not simulators) so XFAIL
them for now for CI and investigate further.
rdar://99981102
Differential Revision: https://reviews.llvm.org/D133963
The fuzzer tests cross_over.test and merge-control-file.test are not handled
correctly on ios device testing. On-device testing requires the macros %t, %s,
etc. to be expanded for a different default directory than when testing on host.
rdar://99889376
Differential Revision: https://reviews.llvm.org/D133811
With this change, fuzz targets may choose to return -1
to indicate that the input should not be added to the corpus
regardless of the coverage it generated.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D128749
As with Linux placce the Counters array in the __libfuzzer_extra_counters
section. This fixes the test on FreeBSD.
Reviewed by: vitalybuka
Differential Revision: https://reviews.llvm.org/D125902
3bd112c720dc fixed the fuzzing test on Linux, which, after
https://reviews.llvm.org/D125933, has one less branch. Turns out, on
Windows, that it still has the extra branch. I'm guessing that's because
exit() isn't known to be noreturn on Windows or something.
Either way, just make the test more tolerant.
https://reviews.llvm.org/D125933 improved some of LLVM's handling of
binary ORs, which meant we have one less conditional branch, because the
'if (Size > 5 && Data[5] == 'R')' and 'if (bits == 63)' branches are now
correctly folded.
This halves the size of LargeTest, dropping time to compile this
file locally from 14s to 5.5s. Hopefully this will also fix the
persistent timeouts in pre-merge checks.
Differential Revision: https://reviews.llvm.org/D124237
This clarifies that this is an LLVM specific variable and avoids
potential conflicts with other projects.
Differential Revision: https://reviews.llvm.org/D119918
These tests appear to be causing timeouts on our silent
Thumbv7 bot: https://lab.llvm.org/staging/#/builders/162/builds/260
It is possible they would complete given enough time. value-profile-switch
seems to take a long time even on a powerful Armv8 machine.
This reverts commit 859ebca744e634dcc89a2294ffa41574f947bd62.
The change contained many unrelated changes and e.g. restored
unit test failes for the old lld port.
This reverts commit 640beb38e7710b939b3cfb3f4c54accc694b1d30.
That commit caused performance degradtion in Quicksilver test QS:sGPU and a functional test failure in (rocPRIM rocprim.device_segmented_radix_sort).
Reverting until we have a better solution to s_cselect_b64 codegen cleanup
Change-Id: Ibf8e397df94001f248fba609f072088a46abae08
Reviewed By: kzhuravl
Differential Revision: https://reviews.llvm.org/D115960
Change-Id: Id169459ce4dfffa857d5645a0af50b0063ce1105
The test has been flaky for years, and I think we should remove it to
eliminate noise on the buildbot.
Neither me nor dokyungs have been able to fully deflake the test, and it
tests a non-default Entropic flag.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D115453
Entropic scheduling with exec-time option can be misled, if inputs
on the right path to become crashing inputs accidentally take more
time to execute before it's added to the corpus. This patch, by letting
more of such inputs added to the corpus (four inputs of size 7 to 10,
instead of a single input of size 2), reduces possibilities of being
influenced by timing flakiness.
A longer-term fix could be to reduce timing flakiness in the fuzzer;
one way could be to execute inputs multiple times and take average of
their execution time before they are added to the corpus.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D113544
I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.
Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105084
Extend the existing single-pass algorithm for `Merger::Merge` with an algorithm that gives better results. This new implementation can be used with a new **set_cover_merge=1** flag.
This greedy set cover implementation gives a substantially smaller final corpus (40%-80% less testcases) while preserving the same features/coverage. At the same time, the execution time penalty is not that significant (+50% for ~1M corpus files and far less for smaller corpora). These results were obtained by comparing several targets with varying size corpora.
Change `Merger::CrashResistantMergeInternalStep` to collect all features from each file and not just unique ones. This is needed for the set cover algorithm to work correctly. The implementation of the algorithm in `Merger::SetCoverMerge` uses a bitvector to store features that are covered by a file while performing the pass. Collisions while indexing the bitvector are ignored similarly to the fuzzer.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105284
