This patch adds the number of executed instructions into the timing
output, which provides more stable results compared to wall or process
time.
The format itself is also tweaked so that it's more amenable for direct
import into a spreadsheet editor.
With inferred modules, the dependency scanner takes care to replace the
fake "__inferred_module.map" path with the file that allowed the module
to be inferred. However, this only worked when such a module was
imported directly in the TU. Whenever such module got loaded
transitively, the scanner would fail to perform the replacement. This is
caused by the fact that PCM files are lossy and drop this information.
This patch makes sure that PCMs include this file for each submodule (in
the `SUBMODULE_DEFINITION` record), fixes one existing test with an
incorrect assertion, and does a little drive-by refactoring of
`ModuleMap`.
This allows the clang driver to know which tool is meant to be executed,
which allows the clang driver to load the right clang config files, and
allows clang to find colocated sysroots.
This makes sure that doing `clang-scan-deps -- <tool> ...` looks up
things in the same way as if one just would execute `<tool> ...`, when
`<tool>` isn't an absolute or relative path.
Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
unclear how that information translates to actual filesystem accesses.
This PR uses `llvm::vfs::TracingFileSystem` to provide that missing
information.
Similar mechanism is implemented for `clang-scan-deps`'s verbose mode
(`-v`). IO contention proved to be a real bottleneck a couple of times
already and this new feature should make those easier to detect in the
future. The tracing VFS is inserted below the caching FS and above the
real FS.
Reverts llvm/llvm-project#102647
I am reverting this change because the `readfile` doesn't actually
perform any useful operation, and yet, for some reason, the test still
passed. This indicates that the modification was unnecessary and could
lead to confusion or unexpected behavior in the future.
This patch updates the clang tests by replacing shell command
substitutions with lit-compatible syntax for setting and referencing
environment variables. Specifically, the use of shell-style variable
substitution (e.g., `DEFAULT_TRIPLE=`and `EXPECTED_RESOURCE_DIR=`) has
been replaced with `env` and `%{env}` to align with lit's internal shell
requirements. These changes ensure that environment variables are
properly set and accessed within the lit environment.
When using the lit internal shell with the command
`LIT_USE_INTERNAL_SHELL=1 ninja check-clang`, one common error
encountered is:
```
FAIL: Clang :: Driver/program-path-priority.c (19 of 20640)
******************** TEST 'Clang :: Driver/program-path-priority.c' FAILED ********************
Exit Code: 127
Command Output (stdout):
--
# RUN: at line 90
DEFAULT_TRIPLE=`/usr/local/google/home/harinidonthula/llvm-project/build/tools/clang/test/Driver/Output/program-path-priority.c.tmp/clang --version | grep "Target:" | cut -d ' ' -f2`
# executed command: 'DEFAULT_TRIPLE=`/usr/local/google/home/harinidonthula/llvm-project/build/tools/clang/test/Driver/Output/program-path-priority.c.tmp/clang' --version
# .---command stderr------------
# | 'DEFAULT_TRIPLE=`/usr/local/google/home/harinidonthula/llvm-project/build/tools/clang/test/Driver/Output/program-path-priority.c.tmp/clang': command not found
# `-----------------------------
# error: command failed with exit status: 127
```
To fix this issue, the patch replaces traditional shell substitutions
with lit's environment variable handling, ensuring compatibility with
the lit internal shell framework. This update applies to both the
handling of the `DEFAULT_TRIPLE` and `EXPECTED_RESOURCE_DIR` variables,
allowing the tests to pass when using the lit internal shell.
The patch also adds `env` to the `PWD` variable setting in the following
command to ensure the environment variable is correctly set within the
lit internal shell:
```
// RUN: %if system-linux %{ env PWD=/proc/self/cwd %clang -### -c --coverage %s -o foo/bar.o 2>&1 | FileCheck --check-prefix=PWD %s %}
```
fixes: #102395
[link to
RFC](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
Previously, since response (.rsp) files weren't expanded at the very
beginning of clang-scan-deps, we only parsed the command-line as
provided in the Clang .cdb file. Unfortunately, when using Unreal
Engine, arguments are always generated in a .rsp file (ie.
`/path/to/clang-cl.exe @/path/to/filename_args.rsp`).
After this patch, `/Fo` can be parsed and added to the final
command-line. Without this option, the make targets that are emitted are
made up from the input file name alone. We have some cases where the
same input in the project generates several output files, so we end up
with duplicate make targets in the scan-deps emitted dependency file.
The dependency scanner only puts top-level affecting module map files on
the command line for explicitly building a module. This is done because
any affecting child module map files should be referenced by the
top-level one, meaning listing them explicitly does not have any meaning
and only makes the command lines longer.
However, a problem arises whenever the definition of an affecting module
lives in a module map that is not top-level. Considering the rules
explained above, such module map file would not make it to the command
line. That's why 83973cf157f7850eb133a4bbfa0f8b7958bad215 started
marking the parents of an affecting module map file as affecting too.
This way, the top-level file does make it into the command line.
This can be problematic, though. On macOS, for example, the Darwin
module lives in "/usr/include/Darwin.modulemap" one of many module map
files included by "/usr/include/module.modulemap". Reporting the parent
on the command line forces explicit builds to parse all the other module
map files included by it, which is not necessary and can get expensive
in terms of file system traffic.
This patch solves that performance issue by stopping marking parent
module map files as affecting, and marking module map files as top-level
whenever they are top-level among the set of affecting files, not among
the set of all known files. This means that the top-level
"/usr/include/module.modulemap" is now not marked as affecting and
"/usr/include/Darwin.modulemap" is.
Pruning non-affecting module maps is useful even when passing module
maps explicitly via `-fmodule-map-file=<path>`. For this situation, this
patch reinstates the behavior we had prior to #87849. For the situation
where the explicit module map file arguments were generated by the
dependency scanner (which already pruned the non-affecting ones), this
patch introduces new `-cc1` flag
`-fno-modules-prune-non-affecting-module-map-files` that avoids the
extra work.
This makes it possible to pass "-o /dev/null" to `clang-scan-deps` and
skip some potentially expensive work, making timings less noisy. Also
removes the need for stream redirection.
The `clang-scan-deps` tool can be used for fast scanning of batches of
compilation commands passed in via the `-compilation-database` option.
This gets awkward in our tests where we have to resort to using
`.in`/`.template` JSON files and running them through `sed` in order to
embed LIT's `%t` variable into them. However, most of our tests only
need to pass single compilation command, so this dance is entirely
unnecessary.
This patch makes sure the existing "per-file" mode (where the
compilation command is passed in-line after the `--` argument) works for
all output formats, not only `P1689`.
This reverts commit 407a2f23 which stopped propagating the callback to module compiles, effectively disabling dependency directive scanning for all modular dependencies. Also added a regression test.
The `ASTWriter` algorithm for computing affecting module maps uses
`SourceManager::translateFile()` to get a `FileID` from a `FileEntry`.
This is slow (O(n)) since the function performs a linear walk over
`SLocEntries` until it finds one with a matching `FileEntry`.
This patch removes this use of `SourceManager::translateFile()` by
tracking `FileID` instead of `FileEntry` in couple of places in
`ModuleMap`, giving `ASTWriter` the desired `FileID` directly. There are
no changes required for clients that still want a `FileEntry` from
`ModuleMap`: the existing APIs internally use `SourceManager` to perform
the reverse `FileID` to `FileEntry` conversion in O(1).
Since b4c83a13f664582015ea22924b9a0c6290d41f5b, `Preprocessor` and
`Lexer` are aware of the concept of scanning dependency directives. This
makes it possible to scan for them on-demand rather than eagerly on the
first filesystem operation (open, or even just stat).
This might improve performance, but is also necessary for the "PCH as
module" mode. Some precompiled header sources use the ".pch" file
extension, which means they were not getting scanned for dependency
directives. This was okay when the PCH was the main input file in a
separate scan step, because there we just lex the file in a
scanning-specific frontend action. But when such source gets treated as
a module implicitly loaded from a TU, it will get compiled as any other
module - with Sema - which will result in compilation errors. (See
attached test case.)
rdar://107663951
Stop overriding -working-directory to CWD during argument parsing, which
should no longer necessary after we set the VFS working directory, and
set FSOpts correctly after parsing arguments so that working-directory
behaves correctly.
It turns out it's not that uncommon for real code to pass a different
set of VFSs while building a PCH than while using the PCH. This can
cause problems as seen in `test/ClangScanDeps/optimize-vfs-pch.m`. If
you scan `compile-commands-tu-no-vfs-error.json` without -Werror and run
the resulting commands, Clang will emit a fatal error while trying to
emit a note saying that it can't find a remapped header.
This also adds textual tracking of VFSs for prebuilt modules that are
part of an included PCH, as the same issue can occur in a module we are
building if we drop VFSs. This has to be textual because we have no
guarantee the PCH had the same list of VFSs as the current TU.
This uses the `PrebuiltModuleListener` to collect `VFSOverlayFiles`
instead of trying to extract it out of a `serialization::ModuleFile`
each time it's needed. There's not a great way to just store a pointer
to the list of strings in the serialized AST.
Canonicalize `-D` and `-U` flags by sorting them and only keeping the
last instance of a given name.
This optimization will only fire if all `-D` and `-U` flags start with a
simple identifier that we can guarantee a simple analysis of can
determine if two flags refer to the same identifier or not. See the
comment on `getSimpleMacroName()` for details of what the issues are.
Previous version of this had issues with sed differences between macOS,
Linux, and Windows. This test doesn't check paths, so just don't run
sed.
Other tests should use `sed -E 's:\\\\?:/:g'` to get portable behavior.
Windows has different command line parsing behavior than Linux for
compilation databases, so the test has been adjusted to ignore that
difference.
Canonicalize `-D` and `-U` flags by sorting them and only keeping the
last instance of a given name.
This optimization will only fire if all `-D` and `-U` flags start with a
simple identifier that we can guarantee a simple analysis of can
determine if two flags refer to the same identifier or not. See the
comment on `getSimpleMacroName()` for details of what the issues are.
Since we already add a `-fmodule-map-file=` argument for every used
modulemap, we can remove all `ModuleMapFiles` entries before adding
them.
This reduces the number of module variants when `-fmodule-map-file=`
appears on the original command line.
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
`OptimizationLevel` and `OptimizeSize` can affect the generated AST. They indirectly affect the `Optimize` and `OptimizeSize` frontend options, which in turn set predefined macro definitions.
This fixes rdar://121228252.
Currently, the dep scanner does not remove LLVM options from the
argument list.
Since LLVM options shouldn't affect the AST, it is safe to remove them
all.
CodeGen options do not affect the AST, so they usually can be ignored.
The only exception to the rule is when a PCM is created with
`-gmodules`.
In that case the Clang module format is switched to object file
container and contains also serialized debug information that can be
affected by debug options. There the following approach was choosen:
1.) Split out all the debug options into a separate `DebugOptions.def`
file. The file is included by `CodeGenOptions.def`, so the change is
transparent to all existing users of `CodeGenOptions.def`.
2.) Reset all CodeGen options, but excluding affecting debug options.
3.) Conditionally reset debug options that can affect the PCM.
This fixes rdar://113135909.
CodeGen options do not affect the AST, so they usually can be ignored.
The only exception to the rule is when a PCM is created with
`-gmodules`.
In that case the Clang module format is switched to object file
container and contains also serialized debug information that can be
affected by debug options. There the following approach was choosen:
1.) Split out all the debug options into a separate `DebugOptions.def`
file. The file is included by `CodeGenOptions.def`, so the change is
transparent to all existing users of `CodeGenOptions.def`.
2.) Reset all CodeGen options, but excluding affecting debug options.
3.) Conditionally reset debug options that can affect the PCM.
This fixes rdar://113135909.
The working directory is included in the PCM, but is not currently part
of the context hash. This causes problems because different builds of a
PCM with exactly the same command line can end up with different binary
content for a PCM. If a build system tracks tasks by both working
directory and command line, it may build a given PCM multiple times,
causing a "module file out of date" error when loading the PCM due to
different sizes.
Both options do not affect the AST content that is serialized into the PCM. This
commit includes the following changes:
1.) Mark `-fvisibility={}` and `-ftype-visibility={}` as benign options.That
means they are no longer considered part of the module hash, which can
reduce the number of module variants.
2.) Add a test to verify the generated LLVM IR is not affected by the default
visibiliy mode in the module.
3.) Add a test to clang-scan-deps to ensure only one module is build, even if
the above mentioned options are used.
This fixes rdar://118246054.
Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's currently only one such
optimization, but more will be added in followup commits.
We prevent translating `#include <FW/PrivateHeader.h>` into an import of
FW_Private when compiling the implementation of FW or FW_Private. This
is specified via `-fmodule-name=` on the TU command line (used to be
`-fmodule-implementation-of`).
This logic is supposed to only kick in when imported directly from a TU,
but it currently also kicks in when compiling the public FW module
explicitly (since it also has `-fmodule-name=` on the command line).
This patch makes sure this logic only kicks in for the case that used to
be `-fmodule-implementation-of` (for the TU), and not for all
`-fmodule-name=` cases (especially for the explicit compile of a
module).
rdar://101051277; related: rdar://37500098&38434694
Previously, Clang wouldn't try to resolve the module for the header
being checked via `__has_include`. This meant that such header was
considered textual (a.k.a. part of the module Clang is currently
compiling).
rdar://116926060
`-include a.h` probes `a.h.pch` and `a.h.gch`, if not found, falls back to
`a.h`. `.pch` is the preferred extension name. Probing .gch is supposed to
provide compatibility with build systems that do
```
clang -x c-header a.h -o out/a.h.gch
clang -include out/a.h -c a.c # out/a.h.gch is present while out/a.h is absent
```
(not sure what projects actually do this with Clang)
But it can often get in the way [^0][^1][^2] when GCC and Clang are mixed as the file
format is incompatible with GCC's. Let's deprecate .gch probing.
Some tests using `-include` are switched to `.pch`.
`test/PCH/pch-dir.c` shows the -Wdeprecated warning.
[^0]: https://discourse.llvm.org/t/how-to-have-clang-ignore-gch-directories/51835
[^1]: https://bugreports.qt.io/browse/QTCREATORBUG-22427
[^2]: https://gitlab.kitware.com/cmake/cmake/-/issues/22081
Previously a relative path would be used as a key for cache lookup and
if the same relative path was used from another compiler invocation with
a different working directory then the first cache entry was erroneously
returned.
This patch solves two problems with the recently added test, which
caused CI failure in an internal downstream system:
1. It connects subsequent `RUN` statements with `&&` into a single
statement; otherwise, the variable defined in the first of them is
out of scope in the subsequent ones where it is used.
2. In our hermetic testing environment, the clang++ binary under test
does not have access to standard include paths (`/usr/include` etc.),
so the test fails because it does not find `stddef.h`. The patch thus
changes the test to use a locally created `a.h` header file.
Differential Revision: https://reviews.llvm.org/D157817
Address https://github.com/llvm/llvm-project/issues/61006.
The actual reason for the issue is about the usage of clang-scan-deps
instead of its functionalities since clang-scan-deps has
already considered the problem before for clang modules.
So this patch tries to document the corresponding solution
and add a test case for it to address the issue.
Add a way to enable -Wsystem-headers only for a specific module. This is
useful for validating a module that would otherwise not see system
header diagnostics without being flooded by diagnostics for unrelated
headers/modules. It's relatively common for a module to be marked
[system] but still wish to validate itself explicitly.
rdar://113401565
Differential Revision: https://reviews.llvm.org/D156948
Allow users to run a dependency scan with a cc1 command line in addition to a driver command line. DependencyScanningAction was already being run with a cc1 command line, but DependencyScanningWorker::computeDependencies assumed that it was always provided a driver command line. Now DependencyScanningWorker::computeDependencies can handle cc1 command lines too.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D156234
