This is a fairly large changeset, but it can be broken into a few
pieces:
- `llvm/Support/*TargetParser*` are all moved from the LLVM Support
component into a new LLVM Component called "TargetParser". This
potentially enables using tablegen to maintain this information, as
is shown in https://reviews.llvm.org/D137517. This cannot currently
be done, as llvm-tblgen relies on LLVM's Support component.
- This also moves two files from Support which use and depend on
information in the TargetParser:
- `llvm/Support/Host.{h,cpp}` which contains functions for inspecting
the current Host machine for info about it, primarily to support
getting the host triple, but also for `-mcpu=native` support in e.g.
Clang. This is fairly tightly intertwined with the information in
`X86TargetParser.h`, so keeping them in the same component makes
sense.
- `llvm/ADT/Triple.h` and `llvm/Support/Triple.cpp`, which contains
the target triple parser and representation. This is very intertwined
with the Arm target parser, because the arm architecture version
appears in canonical triples on arm platforms.
- I moved the relevant unittests to their own directory.
And so, we end up with a single component that has all the information
about the following, which to me seems like a unified component:
- Triples that LLVM Knows about
- Architecture names and CPUs that LLVM knows about
- CPU detection logic for LLVM
Given this, I have also moved `RISCVISAInfo.h` into this component, as
it seems to me to be part of that same set of functionality.
If you get link errors in your components after this patch, you likely
need to add TargetParser into LLVM_LINK_COMPONENTS in CMake.
Differential Revision: https://reviews.llvm.org/D137838
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
This patch changes the PCM serialization logic to refer to input files by their "requested" name. This fixes a bug where the dependency scanner reports the "final" file paths, which can result in failed explicit compiles due to the `module.modulemap` file not being surrounded by the expected framework directory structure.
Depends on D135634.
Reviewed By: benlangmuir, Bigcheese
Differential Revision: https://reviews.llvm.org/D135636
This reverts commit 2f8ac1804827026b44f429dce02730da18a73c50.
After committing a fix for previous buildbot failures in D138970,
re-landing the original change.
This reverts commit 67f34054d6ea8e40fd10cb74441d5ccab004c75b.
This reapplies commit f99e5a9106f08ad92a22c3b114d2052e5c502924.
This improves commit 8ab388e158528d9af5eb0376ef698b243d946f19 that unsucessfully attempted to forward-fix Windows test failure.
This is an attempt to fix a Windows bot failure. In the test introduced in 83973cf1, file dependencies were printed out-of-order (after replacing backslashes with slashes). This might've been caused by styles of some paths being different.
Before the fix the scanning would fail with
`-Werror,-Wnon-modular-include-in-module` despite the warning being
suppressed in the source code.
Existing approach with `-Wno-error` is not sufficient because it negates
only general `-Werror` but not specific `-Werror=...` and some warnings
can still emitted as errors. Make the approach stricter by using `-w`
flag and ignore all warnings, including those upgraded to errors. This
approach is still valid as it doesn't affect the dependencies.
rdar://101588531
Differential Revision: https://reviews.llvm.org/D138252
When a pcm has already been loaded from disk, reuse it from the
InMemoryModuleCache in readASTFileControlBlock. This avoids potentially
reading it again.
As noted in the FIXME, ideally we would also add the module to the cache
if it will be used again later, but that could modify its build state
and we do not have enough context currenlty to know if it's correct.
Differential Revision: https://reviews.llvm.org/D138160
Use a FileEntryRef when retrieving modulemap paths in the scanner so
that we use a path compatible with the original module import, rather
than a FileEntry which can allow unrelated modules to leak paths into
how we build a module due to FileManager mutating the path.
Note: the current change prevents an "unrelated" path, but does not
change how VFS mapped paths are handled (which would be calling
getNameAsRequested) nor canonicalize the path.
Differential Revision: https://reviews.llvm.org/D137989
This is a fix related to D135414. The original intention was to keep `BaseFS` as a member of the worker and conditionally overlay it with local in-memory FS. The `move` of ref-counted `BaseFS` was not intended, and it's a bug.
Disabling parallelism in the "by-module-name" test reliably reproduces this, and the test itself doesn't *need* parallelism. (I think `-j 4` was cargo culted from another test.) Reusing that test to check for correct behavior...
Reviewed By: DavidSpickett
Differential Revision: https://reviews.llvm.org/D136124
The `-ivfsoverlay` flag was only being respected when the scanner was instructed to minimize the inputs. This patch respects that flag even in canonical preprocessing mode.
Depends on D135414.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135416
The dependency scanning worker may create an in-memory VFS and inject that into its `FileManager`. (Implemented in D109485.) This VFS currently persists between scans, which is not correct: in-memory files created in one scan could affect subsequent scans. This patch changes things so that the in-memory VFS is local to `DependencyScanningWorker::computeDependencies()`.
To test this, one would first scan for dependencies of a named module and then run second scan (on the same worker) for something unrelated, which would pick up the in-memory file created in the first scan. This set up is impossible to achieve with `clang-scan-deps`. We could set this up in `ToolingTests`, but the scanner needs to access the physical filesystem to store `.pcm` files. AFAIK we don't have a good way to propagate something like `%t` into unit tests to make that feasible. (This could be achieved with something like the virtualized `OutputManager`.)
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135414
Codegen options are typically unused by modules. Reset some of them to increase sharing between TUs with different flags.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135720
When dep-scanning, canonicalize the module map path as much as we can.
This avoids unnecessarily needing to build multiple versions of a module
due to symlinks or case-insensitive file paths.
Despite the name `tryGetRealPathName`, the previous implementation did
not actually return the realpath most of the time, and indeed it would
be incorrect to do so since the realpath could be outside the module
directory, which would have broken finding headers relative to the
module.
Instead, use a canonicalization that is specific to the needs of
modulemap files (canonicalize the directory separately from the
filename).
Differential Revision: https://reviews.llvm.org/D134923
Update SourceManager::ContentCache::OrigEntry to keep the original
FileEntryRef, and use that to enable ModuleMap::getModuleMapFile* to
return the original FileEntryRef. This change should be NFC for
most users of SourceManager::ContentCache, but it could affect behaviour
for users of getNameAsRequested such as in compileModuleImpl. I have not
found a way to detect that difference without additional functional
changes, other than incidental cases like changes from / to \ on
Windows so there is no new test.
Differential Revision: https://reviews.llvm.org/D135220
This patch provides `FileManager` with the CWD on construction in the worker, rather than later in the action.
Depends on D134976.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134977
This patch removes the ability of a dependency scanning worker to share a `FileManager` instance between individual scans. It's not sound and doesn't provide performance benefits (due to the underlying caching VFS).
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134976
This patch adds new member function to `DependencyScanningWorker` that allows clients to pass custom `DiagnosticConsumer`, and returns `bool`.
This provides more flexibility compared to the existing version that automatically stringifies diagnostics and returns them in `llvm::Error`.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134838
In `ASTWriter`, input files are sorted based on whether they are system or user. The current implementation used single `std::queue` with `push_back` and `push_front`. This resulted in the user files being reversed.
This patch fixes that by keeping the system/user distinction, but otherwise serializing files in the order they were loaded by the `SourceManager`. This is then used in the dependency scanner to report module map dependencies in the correct order.
Depends on D134224.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D134248
Module map files describing excluded headers do affect compilation. Track them in the compiler, serialize them into the PCM file and report them in the scanner.
Depends on D134222.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D134224
This patch fixes compilation failure with explicit modules caused by scanner not reporting the module map describing the module whose implementation is being compiled.
Below is a breakdown of the attached test case. Note the VFS that makes frameworks "A" and "B" share the same header "shared/H.h".
In non-modular build, Clang skips the last import, since the "shared/H.h" header has already been included.
During scan (or implicit build), the compiler handles "tu.m" as follows:
* `@import B` imports module "B", as expected,
* `#import <A/H.h>` is resolved textually (due to `-fmodule-name=A`) to "shared/H.h" (due to the VFS remapping),
* `#import <B/H.h>` is resolved to import module "A_Private", since the header "shared/H.h" is already known to be part of that module, and the import is skipped.
In the end, the only modular dependency of the TU is "B".
In explicit modular build without `-fmodule-name=A`, TU does depend on module "A_Private" properly, not just textually. Clang therefore builds & loads its PCM, and knows to ignore the last import, since "shared/H.h" is known to be part of "A_Private".
But with current scanner behavior and `-fmodule-name=A` present, the last import fails during explicit build. Clang doesn't know about "A_Private" (it's included textually) and tries to import "B_Private" instead, which it doesn't know about either (the scanner correctly didn't report it as dependency). This is fixed by reporting the module map describing "A" and matching the semantics of implicit build.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D134222
The `ScanInstance` is a local variable in `DependencyScanningAction::runInvocation()` that is referenced by `ModuleDepCollector`. Since D132405, `ModuleDepCollector` can escape the function and can outlive its `ScanInstance`. This patch fixes that.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D133988
Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.
Original commit message follows.
---
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
* Factor module map and module file path functions out
* Use a secondary mapping to lookup module deps by ID instead of the
preprocessor module map.
* Sink DirectPrebuiltModularDeps into MDC.
Differential Revision: https://reviews.llvm.org/D132617
The invocation is only ever used to serialize cc1 arguments from, so
instead serialize the arguments inside the dep scanner to simplify the
interface.
Differential Revision: https://reviews.llvm.org/D132616
When compiling a module, its semantics and Clang's behavior are affected by other modules. These modules are typically the **imported** ones. However, during implicit build, some modules end up being compiled and read without being actually imported. This patch starts tracking such modules and serializing them into `.pcm` files. This enables the dependency scanner to construct explicit compilations that mimic implicit build.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D132430
Move copying compiler arguments to a vector<string> and modifying
common module-related options into CompilerInvocation in preparation for
using some of them in more places and to avoid duplicating this code
accidentally in the future.
Differential Revision: https://reviews.llvm.org/D132419
This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.
To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D132066
Instead of delaying the generation of command-lines to after all
dependencies are reported, compute them immediately. This is partly in
preparation for splitting the TU driver command into its constituent cc1
and other jobs, but it also just simplifies working with the compiler
invocation for modules if they are not "without paths".
Also change the computation of the default output path in
clang-scan-deps to scrape the implicit module cache from the
command-line rather than get it from the dependency, since that is now
unavailable at the time we make the callback.
Differential Revision: https://reviews.llvm.org/D131934
Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D131420
Sharing the FileManager across implicit module builds currently leaks
paths looked up in an importer into the built module itself. This can
cause non-deterministic results across scans. It is especially bad for
modules since the path can be saved into the pcm file itself, leading to
stateful behaviour if the cache is shared.
This should not impact the number of real filesystem accesses in the
scanner, since it is already caching in the
DependencyScanningWorkerFilesystem.
Note: this change does not affect whether or not the FileManager is
shared across TUs in the scanner, which is a separate issue.
Differential Revision: https://reviews.llvm.org/D131412
It's an accident that we started return asbolute paths from
FileEntry::getName for all relative paths. Prepare for getName to get
(closer to) return the requested path. Note: conceptually it might make
sense for the dependency scanner to allow relative paths and have the
DependencyConsumer decide if it wants to make them absolute, but we
currently document that it's absolute and I didn't want to change
behaviour here.
Differential Revision: https://reviews.llvm.org/D130934
The "strict context hash" is insufficient to identify module
dependencies during scanning, leading to different module build commands
being produced for a single module, and non-deterministically choosing
between them. This commit switches to hashing the canonicalized
`CompilerInvocation` of the module. By hashing the invocation we are
converting these from correctness issues to performance issues, and we
can then incrementally improve our ability to canonicalize
command-lines.
This change can cause a regression in the number of modules needed. Of
the 4 projects I tested, 3 had no regression, but 1, which was
clang+llvm itself, had a 66% regression in number of modules (4%
regression in total invocations). This is almost entirely due to
differences between -W options across targets. Of this, 25% of the
additional modules are system modules, which we could avoid if we
canonicalized -W options when -Wsystem-headers is not present --
unfortunately this is non-trivial due to some warnings being enabled in
system headers by default. The rest of the additional modules are mostly
real differences in potential warnings, reflecting incorrect behaviour
in the current scanner.
There were also a couple of differences due to `-DFOO`
`-fmodule-ignore-macro=FOO`, which I fixed here.
Since the output paths for the module depend on its context hash, we
hash the invocation before filling in outputs, and rely on the build
system to always return the same output paths for a given module.
Note: since the scanner itself uses an implicit modules build, there can
still be non-determinism, but it will now present as different
module+hashes rather than different command-lines for the same
module+hash.
Differential Revision: https://reviews.llvm.org/D129884
Follow-up to 6626f6fec3d3, this fixes the handling of -MT
* If no targets are provided, we need to invent one since cc1 expects
the driver to have handled it. The default is to use -o, quoting as
necessary for a make target.
* Fix the splitting for empty string, which was incorrectly treated as
{""} instead of {}.
* Add a way to test this behaviour in clang-scan-deps.
Differential Revision: https://reviews.llvm.org/D129607
When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in addition to the pcm
file path. This avoids inheriting per-TU command-line options that
cause non-determinism in the results (non-deterministic command-line for
the module build, non-determinism in which TU's .diag and .d files will
contain the module outputs). In clang-scan-deps we infer whether to
generate dependency or serialized diagnostic files based on an original
command-line. In a real build system this should be modeled explicitly.
Differential Revision: https://reviews.llvm.org/D129389
Dependency scanning does not care about the order of submodules for
correctness, so sort the submodules so that we get the same
command-lines to build the module across different TUs. The order of
inferred submodules can vary depending on the order of #includes in the
including TU.
Differential Revision: https://reviews.llvm.org/D128008
Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for the modules
cache path. This reduces uninteresting differences between
command-lines, which is particularly useful if there is a tool that can
cache the compilations.
Differential Revision: https://reviews.llvm.org/D127883
This fixes the underlying module dependencies, which had a
non-deterministic order, which was also visible in the order of calls to
DependencyConsumer methods. This was not directly observable in
the clang-scan-deps utility, because it was previously seeing a sorted
order from std::map in DependencyScanningTool. However, the underlying
API previously created a likely issue for any other clients. Note: if
you only apply the change from DependencyScanningTool, you can see the
issue in clang-scan-deps, and existing tests will fail
non-deterministicaly.
Differential Revision: https://reviews.llvm.org/D127243
