System make on FreeBSD is missing some GNU make features so out of the
box you get a lot of:
```
make: "<...>/Makefile.rules" line 569: Invalid line type
```
To solve this, you can install gmake which is a port of GNU make.
However because we prefer 'make', gmake won't be used unless you set
LLDB_TEST_MAKE.
To fix that, prefer 'gmake'. Also check (as best we can) that the make
we found is GNU make. This won't be perfect but it's better than the
cryptic error shown above.
```
-- Found make: /usr/bin/make
CMake Warning at /home/ec2-user/llvm-project/lldb/test/API/CMakeLists.txt:63 (message):
'make' tool /usr/bin/make may not be GNU make compatible. Some tests may
fail to build. Provide a GNU compatible 'make' tool by setting
LLDB_TEST_MAKE.
```
When a make isn't found at all, the warning message will show the names
we tried:
```
-- Did not find one of: gmake make
CMake Warning at /home/ec2-user/llvm-project/lldb/test/API/CMakeLists.txt:69 (message):
Many LLDB API tests require a 'make' tool. Please provide it in Path or
pass via LLDB_TEST_MAKE.
```
Bot maintainers should be aware and it became too much of a burden
for developers. In particular on Windows, where make.exe won't be
found in Path typically.
This commit essentially reverts https://reviews.llvm.org/D30453.
In #109961, objcopy util search code was added to dotest.py. dotest.py
should use llvm-X by default if no path to a utility X is provided
externally.
However, it doesn't work out for llvm-objcopy, since objcopy path is
always overridden with the lines being removed here. It causes a problem
with cross-platform testing when objcopy used by cmake doesn't support
targets/executable file formats other than native.
I suppose these lines are unnecessary after #109961, so they can be
safely removed.
Many LLDB's dotest.py based tests require the `make` tool. If it's not found in Path, they fail with an obscure error and show up as `UNRESOLVED`. On Windows, llvm-lit takes care of MSYS based testing tools like cat, printf, etc., but `make` is not part of that. Let's catch the situation early and check for it at configuration time.
This error isn't fatal: It should fail the build, but not immediately stop the configuration process. There might be other issues further down the line that can be caught in the same buildbot run.
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
https://discourse.llvm.org/t/running-lldb-in-a-container/76801/4 found
that the obvious way to use this variable doesn't work, despite what our
docs and examples say.
Perhaps in the past it did but now you need to use ";" as a separator to
make sure the final command line works properly.
For example "-A foo" becomes "-A foo" when python goes to run the runner
script. The script sees this as one command line element, not two. What
you actually want is "-A;foo" which we convert to "-A" "foo" which the
script sees as one option and one value for that option.
The "Script:" printout from dotest is misleading here because it does `"
".join(cmd)` so it looks like it's ok but in fact it's not.
I'm not changing that format though because printing the command as a
Python list is not useful outside of this specific situation.
Add support for the Compact C Type Format (CTF) in LLDB. The format
describes the layout and sizes of C types. It is most commonly consumed
by dtrace.
We generate CTF for the XNU kernel and want to be able to use this in
LLDB to debug kernels for which we don't have dSYMs (anymore). CTF is a
much more limited debug format than DWARF which allows is to be an order
of magnitude smaller: a 1GB dSYM can be converted to a handful of
megabytes of CTF. For XNU, the goal is not to replace DWARF, but rather
to have CTF serve as a "better than nothing" debug info format when
DWARF is not available.
It's worth noting that the LLVM toolchain does not support emitting CTF.
XNU uses ctfconvert to generate CTF from DWARF which is used for
testing.
Differential revision: https://reviews.llvm.org/D154862
A simple sed doing these substitutions:
- `${LLVM_BINARY_DIR}/\$\{CMAKE_CFG_INTDIR}/lib(${LLVM_LIBDIR_SUFFIX})?\>` -> `${LLVM_LIBRARY_DIR}`
- `${LLVM_BINARY_DIR}/\$\{CMAKE_CFG_INTDIR}/bin\>` -> `${LLVM_TOOLS_BINARY_DIR}`
where `\>` means "word boundary".
The only manual modifications were reverting changes in
- `compiler-rt/cmake/Modules/CompilerRTUtils.cmake`
because these were "entry points" where we wanted to tread carefully not not introduce a "loop" which would end with an undefined variable being expanded to nothing.
There are many more occurrences without `CMAKE_CFG_INTDIR`, but those are left for D132316 as they have proved somewhat tricky to fix.
This hopefully increases readability overall, and also decreases the usages of `LLVM_LIBDIR_SUFFIX`, preparing us for D130586.
Reviewed By: sebastian-ne
Differential Revision: https://reviews.llvm.org/D133828
When running LLDB API tests, a user can override test arguments with
LLDB_TEST_USER_ARGS. However, these flags used to be concatenated with a
CMake-derived variable LLDB_TEST_COMMON_ARGS, as below:
```
set(LLDB_DOTEST_ARGS ${LLDB_TEST_COMMON_ARGS};${LLDB_TEST_USER_ARGS}
CACHE INTERNAL STRING)
```
This is problematic, because LLDB_TEST_COMMON_ARGS must be processed
first, while LLDB_TEST_USER_ARGS args must be processed last, so that
user overrides are respected. Currently, if a user attempts to override
one of the "inferred" flags, the user's request is ignored. This is the
case, for example, with `--libcxx-include-dir` and
`--libcxx-library-dir`. Both flags are needed by the greendragon bots.
This commit removes the concatenation above, keeping the two original
variables throughout the entire flow, processing the user's flag last.
The variable LLDB_TEST_COMMON_ARGS needs to be a CACHE property, but it
is modified throughout the CMake file with `set` or `list` or `string`
commands, which don't work with properties. As such, a temporary
variable `LLDB_TEST_COMMON_ARGS_VAR` is created.
This was tested locally by invoking CMake with:
-DLLDB_TEST_USER_ARGS="--libcxx-include-dir=blah --libcxx-library-dir=blah2"
and checking that tests failed appropriately.
Differential Revision: https://reviews.llvm.org/D132642
A simple sed doing these substitutions:
- `${LLVM_BINARY_DIR}/(\$\{CMAKE_CFG_INTDIR}/)?lib(${LLVM_LIBDIR_SUFFIX})?\>` -> `${LLVM_LIBRARY_DIR}`
- `${LLVM_BINARY_DIR}/(\$\{CMAKE_CFG_INTDIR}/)?bin\>` -> `${LLVM_TOOLS_BINARY_DIR}`
where `\>` means "word boundary".
The only manual modifications were reverting changes in
- `compiler-rt/cmake/Modules/CompilerRTUtils.cmake
- `runtimes/CMakeLists.txt`
because these were "entry points" where we wanted to tread carefully not not introduce a "loop" which would end with an undefined variable being expanded to nothing.
This hopefully increases readability overall, and also decreases the usages of `LLVM_LIBDIR_SUFFIX`, preparing us for D130586.
Reviewed By: sebastian-ne
Differential Revision: https://reviews.llvm.org/D132316
When using the system debugserver for testing, copy the binary in the
LLDB.framework Resource directory instead of the build's bin directory.
rdar://82998263
The comment was originally added in 34769d80d. Then D44526
removed the flag added there (but kept the comment), and then
D66966 reintroduced a .noindex dir (which D68606 and then 33fca97880
moved around a bit).
No behavior change.
Differential Revision: https://reviews.llvm.org/D107341
Other LLVM projects use the suffix `-depends` for the test dependencies,
however LLDB uses `-deps` and seems to be the only project under the
LLVM to do so.
In order to make the projects more homogeneous, switch all the
references to `lldb-test-deps` to `lldb-test-depends`.
Additionally, provide a compatibility target with the old name and
depending on the new name, in order to not break anyone workflow.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D102889
The commit got reverted because the tests were being run twice because
of the overlapping test_exec_root. Pavel has since fixed that in
8248dd91d7f042893d4a605e98d19cb1b89a44d4.
Make the API, Shell and Unit tests independent lit test suites. This
allows us to specify different dependencies and skip rebuilding all the
unit test (which is particularly expensive) when running check-lldb-api
or check-lldb-shell.
This does not change the autogenerated targets such as
check-lldb-shell-driver or the top level check-lldb target, which all
continue to work as before.
Differential revision: https://reviews.llvm.org/D98842
This reapplies 7df4eaaa93/D96202, which was reverted due to issues on
windows. These were caused by problems in the computation of the liblldb
directory, which was fixed by D96779.
The original commit message was:
Our test configuration logic assumes that the tests can be run either
with debugserver or with lldb-server. This is not entirely correct,
since lldb server has two "personalities" (platform server and debug
server) and debugserver is only a replacement for the latter.
A consequence of this is that it's not possible to test the platform
behavior of lldb-server on macos, as it is not possible to get a hold of
the lldb-server binary.
One solution to that would be to duplicate the server configuration
logic to be able to specify both executables. However, that seems
excessively redundant.
A well-behaved lldb should be able to find the debug server on its own,
and testing lldb with a different (lldb-|debug)server does not seem very
useful (even in the out-of-tree debugserver setup, we copy the server
into the build tree to make it appear "real").
Therefore, this patch deletes the configuration altogether and changes
the low-level server retrieval functions to be able to both lldb-server
and debugserver paths. They do this by consulting the "support
executable" directory of the lldb under test.
Differential Revision: https://reviews.llvm.org/D96202
Our test configuration logic assumes that the tests can be run either
with debugserver or with lldb-server. This is not entirely correct,
since lldb server has two "personalities" (platform server and debug
server) and debugserver is only a replacement for the latter.
A consequence of this is that it's not possible to test the platform
behavior of lldb-server on macos, as it is not possible to get a hold of
the lldb-server binary.
One solution to that would be to duplicate the server configuration
logic to be able to specify both executables. However, that seems
excessively redundant.
A well-behaved lldb should be able to find the debug server on its own,
and testing lldb with a different (lldb-|debug)server does not seem very
useful (even in the out-of-tree debugserver setup, we copy the server
into the build tree to make it appear "real").
Therefore, this patch deletes the configuration altogether and changes
the low-level server retrieval functions to be able to both lldb-server
and debugserver paths. They do this by consulting the "support
executable" directory of the lldb under test.
Differential Revision: https://reviews.llvm.org/D96202
Replace the dotest command line options and various cmake variables,
which are used for passing the locations of llvm tools to the API tests
with a single variable, which points to the directory these tools are
placed in. Besides reducing repetition, this also makes things more
similar to how "normal" llvm tests are configured.
Differential Revision: https://reviews.llvm.org/D95261
Test runs log some of their output to files inside the LLDB session dir. This
session dir is shared between all tests, so all the tests have to make sure they
choose a unique file name inside that directory. We currently choose by default
`<test-class-name>-<test-method-name>` as the log file name. However, that means
that if not every test class in the test suite has a unique class name, then we
end up with a race condition as two tests will try to write to the same log
file.
I already tried in D83767 changing the format to use the test file basename
instead (which we already require to be unique for some other functionality),
but it seems the code for getting the basename didn't work on Windows.
This patch instead just changes that dotest stores the log files in the build
directory for the current test. We know that directory is unique for this test,
so no need to generate some unique file name now. Also removes all the
environment vars and parameters related to the now unused session dir.
The new log paths now look like this for a failure in 'TestCppOperators`:
```
./lldb-test-build.noindex/lang/cpp/operators/TestCppOperators.test_dwarf/Failure.log
./lldb-test-build.noindex/lang/cpp/operators/TestCppOperators.test_dsym/Failure.log
./lldb-test-build.noindex/lang/cpp/operators/TestCppOperators.test_gmodules/Failure.log
```
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D92498
This patch is a big sed to rename the following variables:
s/PYTHON_LIBRARIES/Python3_LIBRARIES/g
s/PYTHON_INCLUDE_DIRS/Python3_INCLUDE_DIRS/g
s/PYTHON_EXECUTABLE/Python3_EXECUTABLE/g
s/PYTHON_RPATH/Python3_RPATH/g
I've also renamed the CMake module to better express its purpose and for
consistency with FindLuaAndSwig.
Differential revision: https://reviews.llvm.org/D85976
LLVM_TARGET_ARCH is not exported by LLVM so we can't use it from
standalone builds. Default to the architecture in LLVM_HOST_TRIPLE when
no LLDB_DEFAULT_TEST_ARCH was specified.
This reverts commit 29aab9b5c748b28b231e2ca0f9b95453638ade1a.
It seems on Windows the file name is just always "lldbsuite.test.lldbtest" for
all tests and that breaks pretty much everything. Reverting until we have
a better solution.
Summary:
From what I know we already have the restriction that every test in the test
suite needs to have a unique file name as that's used for generating the unique
build directory for a test. It seems there is also a restriction that every test
case class in the test suite needs to have a unique name as that's used to
generate the unique log file name for the test run.
This changes the log file format to use the basename of the test file instead so
that we only have to keep worrying about the 'unique file name' restriction from
now on.
This came up because I started naming the test classes "TestCase" (as repeating
the file name in the test class seems like redudant information that just makes
renaming tests a pain).
Reviewers: labath, JDevlieghere
Reviewed By: labath
Subscribers: mgorny, abidh
Differential Revision: https://reviews.llvm.org/D83767
The Xcode generator does not provide the auto-generated targets where
you can append a folder name to check-lldb. Instead add two custom lit
targets to run just the shell and api tests.
Instead of passing all the arguments for dotest.py as a single CMake
variable, lit now uses separate variables for the different test
binaries. Before this change they'd all get patched as part of the
LLDB_DOTEST_ARGS. We need to patch the new variables as well.
Rather than serializing every argument through LLDB_TEST_COMMON_ARGS, we
can pass some of them directly using their CMake variable. Although this
does introduce some code duplication between lit's site config and the
lldb-dotest utility, it also means that it becomes easier to override
these values (WIP).
Centralize the logic to remove debugserver from
LLVM_DISTRIBUTION_COMPONENTS when LLDB_USE_SYSTEM_DEBUGSERVER is
enabled. Now this happens regardless of whether the tests are enabled.
CMake allows you to set a custom CXX compiler for the API test suite.
However, this variable is never used, because dotest uses the same
compiler to build C and CXX sources.
I'm not sure if this variable was added with the intention of supporting
a different compiler or if this is just a remnant of old functionality.
Given that this hasn't been working for a while, I assume it's safe to
remove.
Differential revision: https://reviews.llvm.org/D69401
The top-level CMake file in the test directory can be simplified by
moving relevant configuration options into the corresponding
subdirectories. Doing so makes it easier to understand what CMake
options are needed by the different test suites.
Differential revision: https://reviews.llvm.org/D69394
LLDB has three major testing strategies: unit tests, tests that exercise
the SB API though dotest.py and what we currently call lit tests. The
later is rather confusing as we're now using lit as the driver for all
three types of tests. As most of this grew organically, the directory
structure in the LLDB repository doesn't really make this clear.
The 'lit' tests are part of the root and among these tests there's a
Unit and Suite folder for the unit and dotest-tests. This layout makes
it impossible to run just the lit tests.
This patch changes the directory layout to match the 3 testing
strategies, each with their own directory and their own configuration
file. This means there are now 3 directories under lit with 3
corresponding targets:
- API (check-lldb-api): Test exercising the SB API.
- Shell (check-lldb-shell): Test exercising command line utilities.
- Unit (check-lldb-unit): Unit tests.
Finally, there's still the `check-lldb` target that runs all three test
suites.
Finally, this also renames the lit folder to `test` to match the LLVM
repository layout.
Differential revision: https://reviews.llvm.org/D68606
llvm-svn: 374184
