This PR unifies the terminology for:
* "context hash" - previously ambiguously referred to as "module hash"
or as overly specific "module context hash"
* "specific module cache path" - previously referred to as just "module
cache path" - hard to distinguish from the command-line-provided module
cache path without the context hash
NFCI
When including a header file, multiple files with the same name may
exist across different search paths, like:
  |-- main.cpp
  |-- **header.h**
  |-- include
  |  └── **header.h**
The compiler usually picks the first match it finds (typically following
MSVC rules for current/include-chain paths first, then regular -I
paths), which may not be the user’s intended header.
This silent behavior can lead to subtle runtime API mismatches or
increase the cost of resolving errors such as “error: use of undeclared
identifier”, especially in large projects.
Therefore, this patch tries to provide a diagnostic message without
changing the current header selection. It does this by performing an
additional search for duplicate filenames across all search paths (both
MSVC rules and standard paths). This informs the user about a potential
"header shadowing" issue and clarifies which header path was actually
used.
Since header searching is much cheaper than file loading, the added
overhead should be within an acceptable range -- assuming the diagnostic
message is valuable.
This PR switches from using `llvm::sys::fs::make_absolute()` to
`FileManager::makeAbsolutePath()` so that `FileSystemOptions` (i.e. the
`-working-directory` option) and the `VFS`'s CWD have a say in how the
prebuilt module paths are resolved. This matches how the rest of the
compiler treats input files.
PR https://github.com/llvm/llvm-project/pull/150123 changed how we
normalize the modules cache path. Unfortunately, empty path would get
normalized to the current working directory. This means that even
explicitly-built PCMs that don't rely on the CWD now embed it, leading
to surprising behavior. This PR fixes that by normalizing an empty
modules cache path to an empty string.
The `llvm::sys::fs::make_absolute(const Twine &, SmallVectorImpl<char>
&)` functions doesn't perform any FS access - it only modifies the
second parameter via path/string operations. This function should live
in the `llvm::sys::path` namespace for consistency and for making it
easier to spot function calls that perform IO.
This PR uses the VFS/`FileManager` to check the system framework marker
instead of going straight to the real file system. This matches the
behavior of other input files of the compiler.
This reverts commit 613caa909c78f707e88960723c6a98364656a926, essentially
reapplying 4a4bddec3571d78c8073fa45b57bbabc8796d13d after moving
`normalizeModuleCachePath` from clangFrontend to clangLex.
This PR is part of an effort to remove file system usage from the
command line parsing code. The reason for that is that it's impossible
to do file system access correctly without a configured VFS, and the VFS
can only be configured after the command line is parsed. I don't want to
intertwine command line parsing and VFS configuration, so I decided to
perform the file system access after the command line is parsed and the
VFS is configured - ideally right before the file system entity is used
for the first time.
This patch delays normalization of the module cache path until
`CompilerInstance` is asked for the cache path in the current
compilation context.
Instead of eagerly populating the `clang::ModuleMap` when looking up a
module by name, this patch changes `HeaderSearch` to only load the
modules that are actually used.
This introduces `ModuleMap::findOrLoadModule` which will load modules
from parsed but not loaded module maps. This cannot be used anywhere
that the module loading code calls into as it can create infinite
recursion.
This currently just reparses module maps when looking up a module by
header. This is fine as redeclarations are allowed from the same file,
but future patches will also make looking up a module by header lazy.
This patch changes the shadow.m test to use explicitly built modules and
`#import`. This test and the shadow feature are very brittle and do not
work in general. The test relied on pcm files being left behind by prior
failing clang invocations that were then reused by the last invocation.
If you clean the cache then the last invocation will always fail. This
is because the input module map and the `-fmodule-map-file=` module map
are parsed in the same module scope, and `-fmodule-map-file=` is
forwarded to implicit module builds. That means you are guaranteed to
hit a module redeclaration error if the TU actually imports the module
it is trying to shadow.
This patch changes when we load A2's module map to after the `A` module
has been loaded, which sets the `IsFromModuleFile` bit on `A`. This
means that A2's `A` is skipped entirely instead of creating a shadow
module, and we get textual inclusion. It is possible to construct a case
where this would happen before this patch too.
An upcoming patch in this series will rework shadowing to work in the
general case, but that's only possible once header -> module lookup is
lazy too.
Now that we have ModuleMapFile.cpp which parses module maps, it's
confusing what ModuleMap::parseModuleMapFile actually does. HeaderSearch
already called this loading a module map, so consistently use that term
in ModuleMap too.
An upcoming patch will allow just parsing a module map without loading
the modules from it.
This makes it so that `CompilerInvocation` can be the only entity that
manages ownership of `HeaderSearchOptions`, making it possible to
implement copy-on-write semantics.
This is to avoid assertion failures like the following when
RedirectingFileSystem's are created and used outside
createVFSFromOverlayFiles.
```
Assertion failed: VFSUsage.size() == getHeaderSearchOpts().VFSOverlayFiles.size() && "A different number of RedirectingFileSystem's were present than " "-ivfsoverlay options passed to Clang!", file S:\SourceCache\llvm-project\clang\lib\Lex\HeaderSearch.cpp, line 162
```
This PR removes the `HeaderFileInfo::Framework` member and reduces the
size of this data type from 32B to 16B. This should improve Clang's
memory usage in situations where it keeps track of lots of header files.
NFCI. Depends on #114459.
This PR removes the `-index-header-map` functionality from Clang. AFAIK
this was only used internally at Apple and is now dead code. The main
motivation behind this change is to enable the removal of
`HeaderFileInfo::Framework` member and reducing the size of that data
structure.
rdar://84036149
I noticed that some PCM files contain `HeaderFileInfo` for headers only
included in a dependent PCM file, which is wasteful.
This patch changes the logic to only write headers that are included
locally. This makes the PCM files smaller and saves some superfluous
deserialization of `HeaderFileInfo` triggered by
`Preprocessor::alreadyIncluded()`.
Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.
This patch stops adjustments of the module cache path beyond what is
done in `ParseHeaderSearchArgs` (making it absolute and removing dots).
This enables more efficient implementation of the caching VFS in
https://github.com/llvm/llvm-project/pull/88800.
Such searches can be costly and non-intuitive. We've seen complaints
from developers that they don't expect clang to find modules on their
own and not in search paths that developers provide. Keeping the search
of modulemaps in subdirectories for code completion as it provides
better user experience.
If you are defining module "UsefulCode" in
"include/UnrelatedName/module.modulemap", it is recommended to rename
the directory "UnrelatedName" to "UsefulCode". If you cannot do so, you
can add to "include/module.modulemap" a line like `extern module
UsefulCode "UnrelatedName/module.modulemap"`, so clang can find module
"UsefulCode" without checking each subdirectory in "include/".
rdar://106677321
---------
Co-authored-by: Jan Svoboda <jan@svoboda.ai>
This patch is helpful to reduce 32 bits for HeaderFileInfo by combining
a uint32_t and pointer into a tagged pointer.
This is reviewed as part of
https://github.com/llvm/llvm-project/pull/92085 and required to be split
as a separate commit
HeaderSearch::MarkFileModuleHeader is no longer properly checking for
no-changes, and so sets the HeaderFileInfo for every `textual header` to
non-external.
When writing out a PCM, we skip serializing headers' `HeaderFileInfo`
struct whenever this condition evaluates to `true`:
```c++
!HFI || (HFI->isModuleHeader && !HFI->isCompilingModuleHeader)
```
However, when Clang parses a module map file, each textual header gets a
`HFI` with `isModuleHeader=false`, `isTextualModuleHeader=true` and
`isCompilingModuleHeader=false`. This means the condition evaluates to
`false` even if the header was never included and the module map did not
affect the compilation. Each PCM file that happened to parse such module
map then contains a copy of the `HeaderFileInfo` struct for all textual
headers, and considers the containing module map affecting.
This patch makes it so that we skip headers that have not been included,
essentially removing the virality of textual headers when it comes to
PCM serialization.
Clang uses the `HeaderFileInfo` struct to track bits of information on
header files, which gets used throughout the compiler. We also use this
to compute the set of affecting module maps in `ASTWriter` and in the
end serialize the information into the `HEADER_SEARCH_TABLE` record of a
PCM file, allowing clients to learn about headers from the module. In
doing so, Clang asks for existing `HeaderFileInfo` for all known
`FileEntries`. Note that this asks the loaded PCM files for the
information they have on each header file in question. This seems
unnecessary: we only want to serialize information on header files that
either belong to the current module or that got included textually.
Loaded PCM files can't provide us with any useful information.
For explicit modules with lazy loading (using `-fmodule-map-file=<path>`
with `-fmodule-file=<name>=<path>`) the compiler knows about header
files listed in the module map files on the command-line. This can be a
large number.
Asking for existing `HeaderFileInfo` can trigger deserialization of
`HEADER_SEARCH_TABLE` from loaded PCM files. Keys of the on-disk hash
table consist of the header file size and modification time. However,
with explicit modules Clang zeroes out the modification time. Moreover,
if you import lots of modules, some of their header files end up having
identical sizes. This means lots of hash collisions that can only be
resolved by running the serialized filename through `FileManager` and
comparing equality of the `FileEntry`. This ends up being super
expensive, essentially re-stating lots of the transitively loaded SDK
header files.
This patch cleans up the API for getting `HeaderFileInfo` and makes sure
`ASTWriter` uses the version that doesn't ask loaded PCM files for more
information. This removes the excessive stat traffic coming from
`ASTWriter` hopefully without changing observable behavior.
Once a file has been `#import`'ed, it gets stamped as if it was `#pragma
once` and will not be re-entered, even on #include. This means that any
errant #import of a file designed to be included multiple times, such as
<assert.h>, will incorrectly mark it as include-once and break the
multiple include functionality. Normally this isn't a big problem, e.g.
<assert.h> can't have its NDEBUG mode changed after the first #import,
but it is still mostly functional. However, when clang modules are
involved, this can cause the header to be hidden entirely.
Objective-C code most often uses #import for everything, because it's
required for most Objective-C headers to prevent double inclusion and
redeclaration errors. (It's rare for Objective-C headers to use macro
guards or `#pragma once`.) The problem arises when a submodule includes
a multiple-include header. The "already included" state is global across
all modules (which is necessary so that non-modular headers don't get
compiled into multiple translation units and cause redeclaration
errors). If another module or the main file #import's the same header,
it becomes invisible from then on. If the original submodule is not
imported, the include of the header will effectively do nothing and the
header will be invisible. The only way to actually get the header's
declarations is to somehow figure out which submodule consumed the
header, and import that instead. That's basically impossible since it
depends on exactly which modules were built in which order.
#import is a poor indicator of whether a header is actually
include-once, as the #import is external to the header it applies to,
and requires that all inclusions correctly and consistently use #import
vs #include. When modules are enabled, consider a header marked
`textual` in its module as a stronger indicator of multiple-include than
#import's indication of include-once. This will allow headers like
<assert.h> to always be included when modules are enabled, even if
#import is erroneously used somewhere.
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
Close https://github.com/llvm/llvm-project/issues/73023
The direct issue of https://github.com/llvm/llvm-project/issues/73023 is
that we entered a header which is marked as pragma once since the
compiler think it is OK if there is controlling macro.
It doesn't make sense. I feel like it should be sufficient to skip it
after we see the '#pragma once'.
From the context, it looks like the workaround is primarily for
ObjectiveC. So we might need reviewers from OC.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
This patch deprecates `module.map` in favor of `module.modulemap`, which
has been the preferred form since 2014. The eventual goal is to remove
support for `module.map` to reduce the number of stats Clang needs to do
while searching for module map files.
This patch touches a lot of files, but the majority of them are just
renaming tests or references to the file in comments or documentation.
The relevant files are:
* lib/Lex/HeaderSearch.cpp
* include/clang/Basic/DiagnosticGroups.td
* include/clang/Basic/DiagnosticLexKinds.td
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
This code was added 17 years ago but never enabled or tested. GCC warns
that -I- is deprecated for them, and Clang gives an error when passed
-I-, so we may as well remove this code rather than hook it up to the
driver and maintain it.
This prevents redefinition errors due to having multiple paths for the
same module map. (rdar://24116019)
Originally implemented and tested downstream by @bcardosolopes, I just
made use of `FileEntryRef::getNameAsRequested()`.
There is a long-standing FIXME in `HeaderSearch.cpp` to use the path separator preferred by the platform instead of forward slash. There was an attempt to fix that (1cf6c28a) which got reverted (cf385dc8). I couldn't find an explanation, but my guess is that some tests assuming forward slash started failing.
This commit fixes tests with that assumption.
This is intended to be NFC, but there are two exceptions to that:
* Some diagnostic messages might now contain backslash instead of forward slash.
* Arguments to the "-remap-file" option that use forward slash might stop kicking in. Separators between potential includer path and header name need to be replaced by backslash in that case.
Fixing Windows buildbot by not using "BuildTemporaries/module.modulemap"
because it is interpreted as defining a module in "BuildTemporaries" directory.
Fix errors like
> module 'MultiPath' is defined in both 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-1352QHUF8RNMU.pcm' and 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-20HNSLLIUDDV1.pcm'
To avoid building extra identical modules `-ivfsoverlay` option is not a
part of the hash like "/3JR48BPRU7BCG/". And it is build system's
responsibility to provide `-ivfsoverlay` options that don't cause
observable differences. We also need to make sure the hash like
"-1352QHUF8RNMU" is not affected by `-ivfsoverlay`. As this hash is
defined by the module map path, use the path prior to any VFS
remappings.
rdar://111921464
Differential Revision: https://reviews.llvm.org/D156749
Fixing Windows buildbot by using the same separators for `-F` and `-I`
paths both in VFS overlay and on command line.
Fix errors like
> module 'MultiPath' is defined in both 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-1352QHUF8RNMU.pcm' and 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-20HNSLLIUDDV1.pcm'
To avoid building extra identical modules `-ivfsoverlay` option is not a
part of the hash like "/3JR48BPRU7BCG/". And it is build system's
responsibility to provide `-ivfsoverlay` options that don't cause
observable differences. We also need to make sure the hash like
"-1352QHUF8RNMU" is not affected by `-ivfsoverlay`. As this hash is
defined by the module map path, use the path prior to any VFS
remappings.
rdar://111921464
Differential Revision: https://reviews.llvm.org/D156749
