This PR removes the `HeaderFileInfo::Framework` member and reduces the
size of this data type from 32B to 16B. This should improve Clang's
memory usage in situations where it keeps track of lots of header files.
NFCI. Depends on #114459.
This PR removes the `-index-header-map` functionality from Clang. AFAIK
this was only used internally at Apple and is now dead code. The main
motivation behind this change is to enable the removal of
`HeaderFileInfo::Framework` member and reducing the size of that data
structure.
rdar://84036149
I noticed that some PCM files contain `HeaderFileInfo` for headers only
included in a dependent PCM file, which is wasteful.
This patch changes the logic to only write headers that are included
locally. This makes the PCM files smaller and saves some superfluous
deserialization of `HeaderFileInfo` triggered by
`Preprocessor::alreadyIncluded()`.
Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.
This patch stops adjustments of the module cache path beyond what is
done in `ParseHeaderSearchArgs` (making it absolute and removing dots).
This enables more efficient implementation of the caching VFS in
https://github.com/llvm/llvm-project/pull/88800.
Such searches can be costly and non-intuitive. We've seen complaints
from developers that they don't expect clang to find modules on their
own and not in search paths that developers provide. Keeping the search
of modulemaps in subdirectories for code completion as it provides
better user experience.
If you are defining module "UsefulCode" in
"include/UnrelatedName/module.modulemap", it is recommended to rename
the directory "UnrelatedName" to "UsefulCode". If you cannot do so, you
can add to "include/module.modulemap" a line like `extern module
UsefulCode "UnrelatedName/module.modulemap"`, so clang can find module
"UsefulCode" without checking each subdirectory in "include/".
rdar://106677321
---------
Co-authored-by: Jan Svoboda <jan@svoboda.ai>
This patch is helpful to reduce 32 bits for HeaderFileInfo by combining
a uint32_t and pointer into a tagged pointer.
This is reviewed as part of
https://github.com/llvm/llvm-project/pull/92085 and required to be split
as a separate commit
HeaderSearch::MarkFileModuleHeader is no longer properly checking for
no-changes, and so sets the HeaderFileInfo for every `textual header` to
non-external.
When writing out a PCM, we skip serializing headers' `HeaderFileInfo`
struct whenever this condition evaluates to `true`:
```c++
!HFI || (HFI->isModuleHeader && !HFI->isCompilingModuleHeader)
```
However, when Clang parses a module map file, each textual header gets a
`HFI` with `isModuleHeader=false`, `isTextualModuleHeader=true` and
`isCompilingModuleHeader=false`. This means the condition evaluates to
`false` even if the header was never included and the module map did not
affect the compilation. Each PCM file that happened to parse such module
map then contains a copy of the `HeaderFileInfo` struct for all textual
headers, and considers the containing module map affecting.
This patch makes it so that we skip headers that have not been included,
essentially removing the virality of textual headers when it comes to
PCM serialization.
Clang uses the `HeaderFileInfo` struct to track bits of information on
header files, which gets used throughout the compiler. We also use this
to compute the set of affecting module maps in `ASTWriter` and in the
end serialize the information into the `HEADER_SEARCH_TABLE` record of a
PCM file, allowing clients to learn about headers from the module. In
doing so, Clang asks for existing `HeaderFileInfo` for all known
`FileEntries`. Note that this asks the loaded PCM files for the
information they have on each header file in question. This seems
unnecessary: we only want to serialize information on header files that
either belong to the current module or that got included textually.
Loaded PCM files can't provide us with any useful information.
For explicit modules with lazy loading (using `-fmodule-map-file=<path>`
with `-fmodule-file=<name>=<path>`) the compiler knows about header
files listed in the module map files on the command-line. This can be a
large number.
Asking for existing `HeaderFileInfo` can trigger deserialization of
`HEADER_SEARCH_TABLE` from loaded PCM files. Keys of the on-disk hash
table consist of the header file size and modification time. However,
with explicit modules Clang zeroes out the modification time. Moreover,
if you import lots of modules, some of their header files end up having
identical sizes. This means lots of hash collisions that can only be
resolved by running the serialized filename through `FileManager` and
comparing equality of the `FileEntry`. This ends up being super
expensive, essentially re-stating lots of the transitively loaded SDK
header files.
This patch cleans up the API for getting `HeaderFileInfo` and makes sure
`ASTWriter` uses the version that doesn't ask loaded PCM files for more
information. This removes the excessive stat traffic coming from
`ASTWriter` hopefully without changing observable behavior.
Once a file has been `#import`'ed, it gets stamped as if it was `#pragma
once` and will not be re-entered, even on #include. This means that any
errant #import of a file designed to be included multiple times, such as
<assert.h>, will incorrectly mark it as include-once and break the
multiple include functionality. Normally this isn't a big problem, e.g.
<assert.h> can't have its NDEBUG mode changed after the first #import,
but it is still mostly functional. However, when clang modules are
involved, this can cause the header to be hidden entirely.
Objective-C code most often uses #import for everything, because it's
required for most Objective-C headers to prevent double inclusion and
redeclaration errors. (It's rare for Objective-C headers to use macro
guards or `#pragma once`.) The problem arises when a submodule includes
a multiple-include header. The "already included" state is global across
all modules (which is necessary so that non-modular headers don't get
compiled into multiple translation units and cause redeclaration
errors). If another module or the main file #import's the same header,
it becomes invisible from then on. If the original submodule is not
imported, the include of the header will effectively do nothing and the
header will be invisible. The only way to actually get the header's
declarations is to somehow figure out which submodule consumed the
header, and import that instead. That's basically impossible since it
depends on exactly which modules were built in which order.
#import is a poor indicator of whether a header is actually
include-once, as the #import is external to the header it applies to,
and requires that all inclusions correctly and consistently use #import
vs #include. When modules are enabled, consider a header marked
`textual` in its module as a stronger indicator of multiple-include than
#import's indication of include-once. This will allow headers like
<assert.h> to always be included when modules are enabled, even if
#import is erroneously used somewhere.
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
Close https://github.com/llvm/llvm-project/issues/73023
The direct issue of https://github.com/llvm/llvm-project/issues/73023 is
that we entered a header which is marked as pragma once since the
compiler think it is OK if there is controlling macro.
It doesn't make sense. I feel like it should be sufficient to skip it
after we see the '#pragma once'.
From the context, it looks like the workaround is primarily for
ObjectiveC. So we might need reviewers from OC.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
This patch deprecates `module.map` in favor of `module.modulemap`, which
has been the preferred form since 2014. The eventual goal is to remove
support for `module.map` to reduce the number of stats Clang needs to do
while searching for module map files.
This patch touches a lot of files, but the majority of them are just
renaming tests or references to the file in comments or documentation.
The relevant files are:
* lib/Lex/HeaderSearch.cpp
* include/clang/Basic/DiagnosticGroups.td
* include/clang/Basic/DiagnosticLexKinds.td
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
This code was added 17 years ago but never enabled or tested. GCC warns
that -I- is deprecated for them, and Clang gives an error when passed
-I-, so we may as well remove this code rather than hook it up to the
driver and maintain it.
This prevents redefinition errors due to having multiple paths for the
same module map. (rdar://24116019)
Originally implemented and tested downstream by @bcardosolopes, I just
made use of `FileEntryRef::getNameAsRequested()`.
There is a long-standing FIXME in `HeaderSearch.cpp` to use the path separator preferred by the platform instead of forward slash. There was an attempt to fix that (1cf6c28a) which got reverted (cf385dc8). I couldn't find an explanation, but my guess is that some tests assuming forward slash started failing.
This commit fixes tests with that assumption.
This is intended to be NFC, but there are two exceptions to that:
* Some diagnostic messages might now contain backslash instead of forward slash.
* Arguments to the "-remap-file" option that use forward slash might stop kicking in. Separators between potential includer path and header name need to be replaced by backslash in that case.
Fixing Windows buildbot by not using "BuildTemporaries/module.modulemap"
because it is interpreted as defining a module in "BuildTemporaries" directory.
Fix errors like
> module 'MultiPath' is defined in both 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-1352QHUF8RNMU.pcm' and 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-20HNSLLIUDDV1.pcm'
To avoid building extra identical modules `-ivfsoverlay` option is not a
part of the hash like "/3JR48BPRU7BCG/". And it is build system's
responsibility to provide `-ivfsoverlay` options that don't cause
observable differences. We also need to make sure the hash like
"-1352QHUF8RNMU" is not affected by `-ivfsoverlay`. As this hash is
defined by the module map path, use the path prior to any VFS
remappings.
rdar://111921464
Differential Revision: https://reviews.llvm.org/D156749
Fixing Windows buildbot by using the same separators for `-F` and `-I`
paths both in VFS overlay and on command line.
Fix errors like
> module 'MultiPath' is defined in both 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-1352QHUF8RNMU.pcm' and 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-20HNSLLIUDDV1.pcm'
To avoid building extra identical modules `-ivfsoverlay` option is not a
part of the hash like "/3JR48BPRU7BCG/". And it is build system's
responsibility to provide `-ivfsoverlay` options that don't cause
observable differences. We also need to make sure the hash like
"-1352QHUF8RNMU" is not affected by `-ivfsoverlay`. As this hash is
defined by the module map path, use the path prior to any VFS
remappings.
rdar://111921464
Differential Revision: https://reviews.llvm.org/D156749
Fix errors like
> module 'MultiPath' is defined in both 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-1352QHUF8RNMU.pcm' and 'path/to/modules.cache/3JR48BPRU7BCG/MultiPath-20HNSLLIUDDV1.pcm'
To avoid building extra identical modules `-ivfsoverlay` option is not a
part of the hash like "/3JR48BPRU7BCG/". And it is build system's
responsibility to provide `-ivfsoverlay` options that don't cause
observable differences. We also need to make sure the hash like
"-1352QHUF8RNMU" is not affected by `-ivfsoverlay`. As this hash is
defined by the module map path, use the path prior to any VFS
remappings.
rdar://111921464
Differential Revision: https://reviews.llvm.org/D156749
- Rename the IsSystem flag to be IsAngled since that's how callers
actually use the flag.
- Since frameworks by convention use <> style includes, make sure
we treat them as Angled
Also update clangd's custom logic for frameworks accordingly.
Differential Revision: https://reviews.llvm.org/D156704
Different requesting modules can have different lookup results, so don't
cache results across modules.
Fixes a regression introduced in reviews.llvm.org/D132779.
Test case based on one provided by Jan Svoboda.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D156000
When Clang loads a PCM that depends on another PCM describing framework module "FW", `ModuleMap` registers "FW" as known, without seeing the module map that defines it (or the adjacent "FW_Private" module map). Later, when looking at a header from "FW_Private", `ModuleMap` returns early due to having knowledge about "FW" and never associates that header with "FW_Private", leading to it being treated as textual. This behavior is caused by D150292, where the scanner stops calling `HeaderSearch::lookupModule()` eagerly for every loaded PCM.
This patch skips an early check when trying to figure out the framework module for a header, which ensures the "FW" and (most importantly) "FW_Private" module maps can be parsed even after loading "FW" from a PCM. Note that the `HeaderSearch::loadModuleMapFile()` function we not call unconditionally has caching behavior of its own, meaning it will avoid parsing module map file repeatedly.
Depends on D150320.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D150478
`HeaderSearch::loadSubdirectoryModuleMaps` `stat`s all the files in a directory which causes the dependency scanning
service to load and cache their contents. This is problematic because a file may be in the process of being generated
and could be cached by the dep-scan service while it is still incomplete.
To address this change `loadSubdirectoryModuleMaps` to ignore regular files.
Differential Revision: https://reviews.llvm.org/D153670
This patch removes some deprecated uses of `{File,Directory}Entry::getName()`. No functional change intended.
Depends on D151854.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D151855
This patch removes some deprecated uses of `{File,Directory}Entry::getName()`. No functional change indended.
Depends on D151853.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D151854
This mimics the `ModuleMap` API and enables D151854, where the `AllowCreation = true` function needs `FileEntryRef` but `AllowCreation = false` functions is happy with plain `FileEntry`. No functional change intended.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D151853
This patch changes the argument type to `HeaderSearch::LookupFile()` from `const DirectoryEntry *` to `DirectoryEntryRef` in order to remove some calls to the deprecated `DirectoryEntry::getName()`.
Depends on D127660.
Reviewed By: bnbarham, benlangmuir
Differential Revision: https://reviews.llvm.org/D127663
This patch changes the return/argument types of `ModuleMap::{load,lookup}ModuleMap()` from `const FileEntry *` to `FileEntryRef` in order to remove uses of the deprecated `DirectoryEntry::getName()`.
Reviewed By: bnbarham
Differential Revision: https://reviews.llvm.org/D127647
