Pruning non-affecting module maps is useful even when passing module
maps explicitly via `-fmodule-map-file=<path>`. For this situation, this
patch reinstates the behavior we had prior to #87849. For the situation
where the explicit module map file arguments were generated by the
dependency scanner (which already pruned the non-affecting ones), this
patch introduces new `-cc1` flag
`-fno-modules-prune-non-affecting-module-map-files` that avoids the
extra work.
It turns out it's not that uncommon for real code to pass a different
set of VFSs while building a PCH than while using the PCH. This can
cause problems as seen in `test/ClangScanDeps/optimize-vfs-pch.m`. If
you scan `compile-commands-tu-no-vfs-error.json` without -Werror and run
the resulting commands, Clang will emit a fatal error while trying to
emit a note saying that it can't find a remapped header.
This also adds textual tracking of VFSs for prebuilt modules that are
part of an included PCH, as the same issue can occur in a module we are
building if we drop VFSs. This has to be textual because we have no
guarantee the PCH had the same list of VFSs as the current TU.
This uses the `PrebuiltModuleListener` to collect `VFSOverlayFiles`
instead of trying to extract it out of a `serialization::ModuleFile`
each time it's needed. There's not a great way to just store a pointer
to the list of strings in the serialized AST.
This patch provides more information to the
`PPCallbacks::InclusionDirective()` hook. We now always pass the
suggested module, regardless of whether it was actually imported or not.
The extra `bool ModuleImported` parameter then denotes whether the
header `#include` will be automatically translated into import the the
module.
The main change is in `clang/lib/Lex/PPDirectives.cpp`, where we take
care to not modify `SuggestedModule` after it's been populated by
`LookupHeaderIncludeOrImport()`. We now exclusively use the `SM`
(`ModuleToImport`) variable instead, which has been equivalent to
`SuggestedModule` until now. This allows us to use the original
non-modified `SuggestedModule` for the callback itself.
(This patch turns out to be necessary for
https://github.com/apple/llvm-project/pull/8011).
Since we already add a `-fmodule-map-file=` argument for every used
modulemap, we can remove all `ModuleMapFiles` entries before adding
them.
This reduces the number of module variants when `-fmodule-map-file=`
appears on the original command line.
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
Currently, the dep scanner does not remove LLVM options from the
argument list.
Since LLVM options shouldn't affect the AST, it is safe to remove them
all.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
The working directory is included in the PCM, but is not currently part
of the context hash. This causes problems because different builds of a
PCM with exactly the same command line can end up with different binary
content for a PCM. If a build system tracks tasks by both working
directory and command line, it may build a given PCM multiple times,
causing a "module file out of date" error when loading the PCM due to
different sizes.
Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's currently only one such
optimization, but more will be added in followup commits.
Now that llvm::support::endianness has been renamed to
llvm::endianness, we can use the shorter form. This patch replaces
llvm::support::endianness with llvm::endianness.
This patch makes the generation of command lines for modular
dependencies lazy/on-demand. That operation is somewhat expensive and
prior to this patch used to be performed multiple times for the
identical `ModuleDeps` (i.e. when they were imported from multiple
different TUs).
We create one `CompilerInvocation` for each modular dependency we
discover. This means we create a lot of copies, even though most of the
invocation is the same between modules. This patch makes use of the
copy-on-write flavor of `CompilerInvocation` to share the common parts,
reducing memory usage and speeding up the scan.
Add a way to enable -Wsystem-headers only for a specific module. This is
useful for validating a module that would otherwise not see system
header diagnostics without being flooded by diagnostics for unrelated
headers/modules. It's relatively common for a module to be marked
[system] but still wish to validate itself explicitly.
rdar://113401565
Differential Revision: https://reviews.llvm.org/D156948
The current `ASTReader::visitInputFiles()` function calls into `FileManager` to create `FileEntryRef` objects. This ends up being fairly costly in `clang-scan-deps`, where we mostly only care about file paths.
This patch introduces new `ASTReader` API that gives clients access to just the serialized paths. Since the scanner needs both the as-requested path and the on-disk one (and doesn't want to transform the former into the latter via `FileManager`), this patch starts serializing both of them into the PCM file if they differ.
This increases the size of scanning PCMs by 0.1% and speeds up scanning by 5%.
Reviewed By: benlangmuir, vsapsai
Differential Revision: https://reviews.llvm.org/D157066
This patch makes use of the infrastructure established in D157046 to speed up computation of the canonical context hash in the dependency scanner. This is somewhat hot code, since it's ran for all modules in the dependency graph of every TU.
I also tried an alternative approach that tried to avoid allocations as much as possible (essentially doing `HashBuilder.add(Arg.toStringRef(ArgVec))`), but that turned out to be slower than approach in this patch.
Note that this is not problematic in the same way command-line hashing used to be prior D143027. The lambda is now being called even for constant strings.
Depends on D157046.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D157052
This information is already exposed via `TranslationUnitDeps::ClangModuleDeps` on the `DependencyScanningTool` level, and this patch also adds it on the `DependencyScanningWorker` level via `DependencyConsumer::handleDirectModuleDependency()`.
Besides being redundant, this bit of information is misleading for clients that share single `ModuleDeps` instance between multiple TUs (by using the `AlreadySeen` set). The module can be imported directly in some TUs but transitively in others.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D156563
In file `clang/lib/Basic/Module.cpp` the `Module` class had `submodule_begin()` and `submodule_end()` functions to retrieve corresponding iterators for private vector of Modules. This commit removes mentioned functions, and replaces all of theirs usages with `submodules()` function and range-based for-loops.
Differential Revision: https://reviews.llvm.org/D148954
When not performing codegen, we can strip the coverage-data-file and
coverage-notes-file options to improve canonicalization.
rdar://107443796
Differential Revision: https://reviews.llvm.org/D147282
The idea is to split the callbacks that are used to consume dependency
information (DependencyConsumer) from callbacks that modify the scan
behaviour itself in any way (DependencyActionController). Currently this
is just lookupModuleOutput, but we have additional callbacks related to
CAS support that we intend to upstream in the future.
Differential Revision: https://reviews.llvm.org/D144058
LexedFileChanged has the semantics we want of ignoring #line/etc. It's
also consistent with other dep collectors like DependencyFileGenerator.
Differential Revision: https://reviews.llvm.org/D143613
We were not hashing constant strings in the command-line, only ones that
required allocations. This was causing us to get the same hash across
different flag options.
rdar://101053855
Differential Revision: https://reviews.llvm.org/D143027
The -arcmt-action= and -objcmt-migrate* actions were being passed to
module builds. This caused these builds to fail, as they are secondary
actions that suppress emitting modules.
Differential Revision: https://reviews.llvm.org/D143040
Use the name "as requested" for the path of the implemented module's
modulemap file, just as we do for other modulemap file paths. This fixes
fatal errors with modules where we tried to find framework headers
relative to the wrong directory when imported by an implementation file
of the same module.
rdar://104619123
Differential Revision: https://reviews.llvm.org/D142501
This patch removes some dead code in the dependency scanner.
The `ModuleDeps::ImplicitModulePCMPath` member stopped being used in D131934.
The strict context hash was replaced in D129884 by hash of the canonical command line.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D142416
In D106100, we started guarding against spurious dependencies on modules that ended up being textual includes and thus didn't have any AST file associated. That patch accounted only for direct dependencies. There's a way how to get spurious dependencies for modules that are transitive. This patch guards against that scenario and adds a test case.
(Note that since D142167, we don't allow `@import FW_Private` with `-fmodule-name=FW` anymore. However, that check lives in sema, which the scanner doesn't run. Being defensive in this patch therefore still makes sense.)
rdar://104324602
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D142165
I originally thought we needed to add module file inputs for modular
deps at the same time as outputs because they depend on the
lookupModuleOutput callback, but this is not the case: they only depend
on the callback results for other modules, which have already been
computed by this point. So move them earlier so that they're set in the
CompilerInvocation at the same time as other inputs. This makes the
code easier to understand.
This change is effectively NFC, though it technically changes the module
exact value of the context hash.
Differential Revision: https://reviews.llvm.org/D142392
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
This patch changes the PCM serialization logic to refer to input files by their "requested" name. This fixes a bug where the dependency scanner reports the "final" file paths, which can result in failed explicit compiles due to the `module.modulemap` file not being surrounded by the expected framework directory structure.
Depends on D135634.
Reviewed By: benlangmuir, Bigcheese
Differential Revision: https://reviews.llvm.org/D135636
This reverts commit 67f34054d6ea8e40fd10cb74441d5ccab004c75b.
This reapplies commit f99e5a9106f08ad92a22c3b114d2052e5c502924.
This improves commit 8ab388e158528d9af5eb0376ef698b243d946f19 that unsucessfully attempted to forward-fix Windows test failure.
This is an attempt to fix a Windows bot failure. In the test introduced in 83973cf1, file dependencies were printed out-of-order (after replacing backslashes with slashes). This might've been caused by styles of some paths being different.
Use a FileEntryRef when retrieving modulemap paths in the scanner so
that we use a path compatible with the original module import, rather
than a FileEntry which can allow unrelated modules to leak paths into
how we build a module due to FileManager mutating the path.
Note: the current change prevents an "unrelated" path, but does not
change how VFS mapped paths are handled (which would be calling
getNameAsRequested) nor canonicalize the path.
Differential Revision: https://reviews.llvm.org/D137989
Codegen options are typically unused by modules. Reset some of them to increase sharing between TUs with different flags.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135720
When dep-scanning, canonicalize the module map path as much as we can.
This avoids unnecessarily needing to build multiple versions of a module
due to symlinks or case-insensitive file paths.
Despite the name `tryGetRealPathName`, the previous implementation did
not actually return the realpath most of the time, and indeed it would
be incorrect to do so since the realpath could be outside the module
directory, which would have broken finding headers relative to the
module.
Instead, use a canonicalization that is specific to the needs of
modulemap files (canonicalize the directory separately from the
filename).
Differential Revision: https://reviews.llvm.org/D134923
Update SourceManager::ContentCache::OrigEntry to keep the original
FileEntryRef, and use that to enable ModuleMap::getModuleMapFile* to
return the original FileEntryRef. This change should be NFC for
most users of SourceManager::ContentCache, but it could affect behaviour
for users of getNameAsRequested such as in compileModuleImpl. I have not
found a way to detect that difference without additional functional
changes, other than incidental cases like changes from / to \ on
Windows so there is no new test.
Differential Revision: https://reviews.llvm.org/D135220
In `ASTWriter`, input files are sorted based on whether they are system or user. The current implementation used single `std::queue` with `push_back` and `push_front`. This resulted in the user files being reversed.
This patch fixes that by keeping the system/user distinction, but otherwise serializing files in the order they were loaded by the `SourceManager`. This is then used in the dependency scanner to report module map dependencies in the correct order.
Depends on D134224.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D134248
