This patch removes getVirtualFile because it has been deprecated for
more than 10 months since:
commit b1aea98cfa357e23f4bb52232da5f41781f23bff
Author: Jan Svoboda <jan_svoboda@apple.com>
Date: Wed Sep 25 10:36:44 2024 -0700
I'm not aware of any downstream use AFAICT.
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
See https://github.com/llvm/llvm-project/issues/126629 . Right now they
are not supported properly and support requires modifying several layers
of LLVM. For now simply reject them while proper support is being
developed to avoid potential security problems.
This PR removes the `FileManager` APIs that have been deprecated for a
while.
LLVM 20.1.0 that was released earlier this month contains the formal
deprecation of these APIs, so these should be fine to remove in the next
major release.
Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:
// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>
I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.
Note that PointerUnion::{is,get} have been soft deprecated in
PointerUnion.h:
// FIXME: Replace the uses of is(), get() and dyn_cast() with
// isa<T>, cast<T> and the llvm::dyn_cast<T>
I'm not touching PointerUnion::dyn_cast for now because it's a bit
complicated; we could blindly migrate it to dyn_cast_if_present, but
we should probably use dyn_cast when the operand is known to be
non-null.
This patch adds an IsText parameter to the following getBufferForFile,
getBufferForFileImpl. We introduce a new virtual function
openFileForReadBinary which defaults to openFileForRead except in
RealFileSystem which uses the OF_None flag instead of OF_Text.
The default is set to OF_Text instead of OF_None, this change in value
does not affect any other platforms other than z/OS. Setting this
parameter correctly is required to open files on z/OS in the correct
encoding. The IsText parameter is based on the context of where we open
files, for example, in the ASTReader, HeaderMap requires that files
always be opened in binary even though they might be tagged as text.
This patch adds an IsText parameter to the following functions
openFileForRead, getBufferForFile, getBufferForFileImpl and determines
whether a file is text by querying the file tag on z/OS. The default is
set to OF_Text instead of OF_None, this change in value does not affect
any other platforms other than z/OS.
Clang's `-cc1 -print-stats` shows lots of useful internal data including
basic `FileManager` stats. Since this layer caches some results, it is
unclear how that information translates to actual filesystem accesses.
This PR uses `llvm::vfs::TracingFileSystem` to provide that missing
information.
Similar mechanism is implemented for `clang-scan-deps`'s verbose mode
(`-v`). IO contention proved to be a real bottleneck a couple of times
already and this new feature should make those easier to detect in the
future. The tracing VFS is inserted below the caching FS and above the
real FS.
This commit implements the entirety of the now-accepted [N3017
-Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded by
AST consumers (CodeGen or constant expression evaluators) or expand
embed directive as a comma expression.
This reverts commit
682d461d5a.
---------
Co-authored-by: The Phantom Derpstorm <phdofthehouse@gmail.com>
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>
This commit implements the entirety of the now-accepted [N3017 -
Preprocessor
Embed](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3017.htm) and
its sister C++ paper [p1967](https://wg21.link/p1967). It implements
everything in the specification, and includes an implementation that
drastically improves the time it takes to embed data in specific
scenarios (the initialization of character type arrays). The mechanisms
used to do this are used under the "as-if" rule, and in general when the
system cannot detect it is initializing an array object in a variable
declaration, will generate EmbedExpr AST node which will be expanded
by AST consumers (CodeGen or constant expression evaluators) or
expand embed directive as a comma expression.
---------
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
Co-authored-by: H. Vetinari <h.vetinari@gmx.com>
Co-authored-by: Podchishchaeva, Mariya <mariya.podchishchaeva@intel.com>
The `FileManager` might create a virtual directory that can be used
later as a search path. This is the case when we use remapping, as
demonstrated in the suggested LIT test.
We might encounter a 'false cache miss' and add the same directory
twice into `FileManager::SeenDirEntries` if the added record is a real
directory that is present on a disk:
- Once as a virtual directory
- And once as a real one
This isn't a problem if the added directories have the same name, as in
this case, we will get a cache hit. However, it could lead to
compilation errors if the directory names are different but point to the
same folder. For example, one might use an absolute name and another a
relative one. For instance, the **implicit-module-remap.cpp** LIT test
will fail with the following message:
```
/.../implicit-module-remap.cpp.tmp/test.cpp:1:2: fatal error: module 'a'
was built in directory '/.../implicit-module-remap.cpp.tmp' but now
resides in directory '.'
1 | #include "a.h"
| ^
1 error generated.
```
The suggested fix checks if the added virtual directory is present on
the disk and handles it as a real one if that is the case.
We recently implemented a new option allowing relinking of bitcode
modules via the "-mllvm -relink-builtin-bitcode-postop"
option.
This implementation relied on llvm::CloneModule() in order to pass
copies to modules and preserve the original modules for later relinking.
However, cloning modules has been found to be prohibitively expensive,
significantly increasing compilation time for large bitcode libraries.
In this patch, we shift the relink option implementation to instead link
the original modules initially, and reload modules from the file system
if relinking is requested. This approach results in significantly
reduced overhead.
We accomplish this by creating a new ReloadModules() routine that can be
called from a BackendConsumer class, to mimic the behavior of
ASTConsumer's loadLinkModules(), but without access to the
CompilerInstance.
Because loading the bitcodes from the filesystem requires access to the
FileManager class, we also forward a reference to the CompilerInstance
class to the BackendConsumer. This mirrors what is already done for
several CompilerInstance members, such as TargetOptions and
CodeGenOptions.
Finally, we needed to add a const specifier to the
FileManager::getBufferForFile() routine to allow it to be called using
the const reference returned from CompilerInstance::getFileManager()
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
The files and directories that Clang accesses are uniqued by their
inode. For each inode `FileManager` will create exactly one `FileEntry`
or `DirectoryEntry` object, which makes answering the question _"Are
these two files/directories the same?"_ a simple pointer equality check.
However, since the same inode can be accessed through multiple different
paths, asking the `FileEntry` or `DirectoryEntry` object _"What is your
name?"_ doesn't have clear semantics. In c0ff9908 we started reporting
the most recent name used to access the entry, which turned out to be
necessary for Clang modules. However, the long-term solution has always
been to explicitly track the as-requested name. This has been
implemented in 4dc5573a as `FileEntryRef` and `DirectoryEntryRef`.
The `DirectoryEntry::getName()` interface has been deprecated since the
Clang 17 release and `FileEntry::getName()` since Clang 18. We have
replaced uses of these deprecated APIs in `main` with
`DirectoryEntryRef::getName()` and `FileEntryRef::getName()`
respectively.
This makes it possible to remove `{File,Directory}Entry::getName()` for
good along with the `FileManager` code that implements them.
This patch adopts `FileEntryRef` in the `HeaderFileInfo`-writing part of
`ASTWriter`.
First, this patch removes the loop over
`FileManager::VirtualFileEntries`. It's redundant, since all virtual
file entries are also present in `SeenFileEntries` and thus already in
`UIDToFiles`.
Second, since we now no longer rely on
`FileEntry::getLastRef()`/`FileEntry::getName()`, this patch takes care
to establish which path gets used for each UID by picking the
`FileEntryRef` with the most "`<`" name (instead of just relying on the
`StringMap` iteration order).
Note that which `FileEntry`/`FileEntryRef` objects we pick for each UID
for serialization into the `llvm::OnDiskChainedHashTable` doesn't really
matter. The hash function only includes the file size and modification
time. The file name only plays role during resolution of hash
collisions, in which case it goes through `FileManager` and resolves to
a `FileEntry` that gets pointer-compared with the queried `FileEntry`.
(Reincarnation of [D143414](https://reviews.llvm.org/D143414) and
[D142780](https://reviews.llvm.org/D142780).)
Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.
Changes lit logic to:
Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.
Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.
Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).
How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows
Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir
Patch by Tristan Labelle <tristan@thebrowser.company>!
This patch removes the last use of deprecated `DirectoryEntry::getName()`.
Depends on D151855.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D151922
For redirected file entries, `FileEntryRef::getDir()` returns the parent directory entry of the target file entry. This differs from `FileEntry::getDir()` that always returns the parent directory that was last used to look up that file.
After switching from `FileEntry` to `FileEntryRef` for umbrella headers in D142113, this discrepancy became observable and caused Clang to emit incorrect diagnostics.
This patch changes Clang so that it always associates `FileEntryRef` with the parent directory that was used to look it up. This brings its behavior closer to `FileEntry`, but without the hacky mutation.
This also ensures that `llvm::sys::path::parent_path(FileRef->getNameAsRequested()) == FileRef->getDir()->getName()`. Previously, `FileRef->getDir()` would fall underneath the redirecting VFS into the world of on-disk paths.
Reviewed By: benlangmuir, rmaz
Differential Revision: https://reviews.llvm.org/D151398
We no longer support GCC 5.1 (D122976), so let's remove the workaround in FileEntry.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D143427
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
In 6a79e2ff1989b we changed Filemanager::getEntryRef() to return the
redirecting FileEntryRef instead of looking through the redirection.
This commit fixes the case when looking up a cached file path to also
return the redirecting FileEntryRef. This mainly affects the behaviour
of calling getNameAsRequested() on the resulting entry ref.
Differential Revision: https://reviews.llvm.org/D131273
As progress towards having FileManager::getFileRef() return the path
as-requested by default, return a FileEntryRef that can use
getNameAsRequested() to retrieve this path, with the ultimate goal that
this should be the behaviour of getName() and clients should explicitly
request the "external" name if they need to (see comment in
FileManager::getFileRef). For now, getName() continues to return the
external path by looking through the redirects.
For now, the new function is only used in unit tests.
Differential Revision: https://reviews.llvm.org/D131004
As progress towards having FileEntryRef contain the requested name of
the file, this commit narrows the "remap" hack to only apply to paths
that were remapped to an external contents path by a VFS. That was
always the original intent of this code, and the fact it was making
relative paths absolute was an unintended side effect.
Differential Revision: https://reviews.llvm.org/D130935
If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.
Resolves rdar://90578880 and llvm-project#53306.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D123398
- isValid: FileManager only ever returns valid FileEntries (see next point)
- construction from outside FileManager (both FileEntry and DirectoryEntry).
It's not possible to create a useful FileEntry this way, there are no setters.
This was only used in FileEntryTest, added a friend to enable this.
A real constructor is cleaner but requires larger changes to FileManager.
Differential Revision: https://reviews.llvm.org/D123197
This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstances. Specifically,
since crash reproducers have `UseExternalNames` set to false, the
`File->getFileEntry().getDir()->getName()` call in `DoFrameworkLookup`
would use the *cached* directory name instead of the directory of the
looked-up file.
The plan is to re-commit this patch but to *add*
`ExposesExternalVFSPath` rather than replace `IsVFSMapped`.
Differential Revision: https://reviews.llvm.org/D123103
If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
This also renames `IsVFSMapped` to `ExposesExternalVFSPath` and only
sets it if `UseExternalName` is true. This flag then represents that the
`Status` has an external path that's different from its virtual path.
Right now the contained path is still the external path, but further PRs
will change this to *always* be the virtual path. Clients that need the
external can then request it specifically.
Note that even though `ExposesExternalVFSPath` isn't set for all
VFS-mapped paths, `IsVFSMapped` was only being used by a hack in
`FileManager` that was specific to module searching. In that case
`UseExternalNames` is always `true` and so that hack still applies.
Resolves rdar://90578880 and llvm-project#53306.
Differential Revision: https://reviews.llvm.org/D122549
Use the new sys::path::is_style_posix() and is_style_windows() in a few
places that need to detect the system's native path style.
In llvm/lib/Support/Path.cpp, this patch removes most uses of the
private `real_style()`, where is_style_posix() and is_style_windows()
are just a little tidier.
Elsewhere, this removes `_WIN32` macro checks. Added a FIXME to a
FileManagerTest that seemed fishy, but maintained the existing
behaviour.
Differential Revision: https://reviews.llvm.org/D112289
Document 'use-external-name' and the various bits of logic that make it
work, to avoid others having to repeat the archival work (given that I
added getFileRefReturnsCorrectNameForDifferentStatPath to
FileManagerTest, seems possible I understood this once before!).
- b59cf679e81483cbb3a9252056b7528f4c49586c added 'use-external-name' to
RedirectingFileSystem. This causes `stat`s to return the external
name for a redirected file instead of the name it was accessed by,
leaking it through the VFS.
- d066d4c849be06a01c0d17e8dc206913f4e7bfe3 propagated the external name
further through clang::FileManager.
- 4dc5573acc0d2e7c59d8bac2543eb25cb4b32984, which added
clang::FileEntryRef to clang::FileManager, has complicated concession
to account for this as well (since refactored a bit).
The goal of 'use-external-name' is to enable Clang to report "real" file
paths to users (via diagnostics) and to external tools (such as
debuggers reading debug info and build systems reading `.d` files).
I've added FIXMEs to look at other channels for communicating the
external names, since the current implementation adds complexity to
FileManager and exposes an inconsistent interface to clients.
Besides that, the FileManager logic appears to be kicking in outside of
'use-external-name'. Seems that *some* vfs::FileSystem implementations
canonicalize some paths returned by `stat` in *some* cases (the bug
isn't fully understood yet). Volodymyr Sapsai is investigating, this at
least better documents what *is* understood.
C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)
This is mostly a mechanical change, but a testcase that contains
parts of the StringRef class (clang/test/Analysis/llvm-conventions.cpp)
isn't touched.
After https://reviews.llvm.org/D90484 libclang is unable to read a serialized diagnostic file
which contains a diagnostic which came from a file with an empty filename. The reason being is
that the serialized diagnostic reader is creating a virtual file for the "" filename, which now
fails after the changes in https://reviews.llvm.org/D90484. This patch restores the previous
behavior in getVirtualFileRef by allowing it to construct a file entry ref with an empty name by
pretending its name is "." so that the directory entry can be created.
Differential Revision: https://reviews.llvm.org/D100428
Add support for stdin to SourceManager and FileManager. Adds
FileManager::getSTDIN, which adds a FileEntryRef for `<stdin>` and reads
the MemoryBuffer, which is stored as `FileEntry::Content`.
Eventually the other buffers in `ContentCache` will sink to here as well
-- we probably usually want to load/save a MemoryBuffer eagerly -- but
it's happening early for stdin to get rid of
CompilerInstance::InitializeSourceManager's final call to
`SourceManager::overrideFileContents`.
clang/test/CXX/modules-ts/dcl.dcl/dcl.module/dcl.module.export/p1.cpp
relies on building a module from stdin; supporting that requires setting
ContentCache::BufferOverridden.
Differential Revision: https://reviews.llvm.org/D93148
Handle named pipes natively in SourceManager and FileManager, removing a
call to `SourceManager::overrideFileContents` in
`CompilerInstance::InitializeSourceManager` (removing a blocker for
sinking the content cache to FileManager (which will incidently sink
this new named pipe logic with it)).
SourceManager usually checks if the file entry's size matches the
eventually loaded buffer, but that's now skipped for named pipes since
the `stat` won't reflect the full size. Since we can't trust
`ContentsEntry->getSize()`, we also need shift the check for files that
are too large until after the buffer is loaded... and load the buffer
immediately in `createFileID` so that no client gets a bad value from
`ContentCache::getSize`. `FileManager::getBufferForFile` also needs to
treat these files as volatile when loading the buffer.
Native support in SourceManager / FileManager means that named pipes can
also be `#include`d, and clang/test/Misc/dev-fd-fs.c was expanded to
check for that.
This is a new version of 3b18a594c7717a328c33b9c1eba675e9f4bd367c, which
was reverted in b34632201987eed369bb7ef4646f341b901c95b8 since it was
missing the `SourceManager` changes.
Differential Revision: https://reviews.llvm.org/D92531
This reverts commit 3b18a594c7717a328c33b9c1eba675e9f4bd367c, since
apparently this doesn't work everywhere. E.g.,
clang-x86_64-debian-fast/3889
(http://lab.llvm.org:8011/#/builders/109/builds/3889) gives me:
```
+ : 'RUN: at line 8'
+ /b/1/clang-x86_64-debian-fast/llvm.obj/bin/clang -x c /dev/fd/0 -E
+ cat /b/1/clang-x86_64-debian-fast/llvm.src/clang/test/Misc/dev-fd-fs.c
fatal error: file '/dev/fd/0' modified since it was first processed
1 error generated.
```
Remove compilicated logic from CompilerInstance::InitializeSourceManager
to deal with named pipes, updating FileManager::getBufferForFile to
handle it in a more straightforward way. The existing test at
clang/test/Misc/dev-fd-fs.c covers the new behaviour (just like it did
the old behaviour).
Differential Revision: https://reviews.llvm.org/D90733
