When we lookup in the external filesystem, do not remove . and ..
components from the original path. For .. this is a correctness issue in
the presence of symlinks, while for . it is simply better practice to
preserve the original path to better match the behaviour of other
filesystems. The only modification we need is to apply the working
directory, since it could differ from the external filesystem.
rdar://123655660
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
The files and directories that Clang accesses are uniqued by their
inode. For each inode `FileManager` will create exactly one `FileEntry`
or `DirectoryEntry` object, which makes answering the question _"Are
these two files/directories the same?"_ a simple pointer equality check.
However, since the same inode can be accessed through multiple different
paths, asking the `FileEntry` or `DirectoryEntry` object _"What is your
name?"_ doesn't have clear semantics. In c0ff9908 we started reporting
the most recent name used to access the entry, which turned out to be
necessary for Clang modules. However, the long-term solution has always
been to explicitly track the as-requested name. This has been
implemented in 4dc5573a as `FileEntryRef` and `DirectoryEntryRef`.
The `DirectoryEntry::getName()` interface has been deprecated since the
Clang 17 release and `FileEntry::getName()` since Clang 18. We have
replaced uses of these deprecated APIs in `main` with
`DirectoryEntryRef::getName()` and `FileEntryRef::getName()`
respectively.
This makes it possible to remove `{File,Directory}Entry::getName()` for
good along with the `FileManager` code that implements them.
D152495 makes clang warn on unused variables that are declared in conditions like `if (int var = init) {}`
This patch is an NFC fix to suppress the new warning in llvm,clang,lld builds to pass CI in the above patch.
Differential Revision: https://reviews.llvm.org/D158016
There is a libstdc++ stl_map.h bug that is only back ported to 7.5.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78595#c13
```
/tmp/opt/gcc-7.3.0/include/c++/7.3.0/bits/stl_tree.h:2091:28: error: no matching function for call to ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, std::unique_ptr<llvm::vfs::detail::InMemoryNode> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, std::unique_ptr<llvm::vfs::detail::InMemoryNode> > >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, std::unique_ptr<llvm::vfs::detail::InMemoryNode> > > >::_M_get_insert_unique_pos(std::pair<llvm::StringRef, std::unique_ptr<llvm::vfs::detail::InMemoryNode> >::first_type&)’
= _M_get_insert_unique_pos(_KeyOfValue()(__v));
~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
```
Just construct a std::string from StringRef to work around it.
A follow-up to D135841. This patch returns the virtual path for directories from `RedirectingFileSystem`. This ensures the contents of `Path` are the same as the contents of `FS->getRealPath(Path)`. This also means we can drop the workaround in Clang's module map canonicalization, where we couldn't use the real path for a directory if it resolved to a different `DirectoryEntry`. In addition to that, we can also avoid introducing new workaround for a bug triggered by the newly introduced test case.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D135849
In createPhysicalFileSystem, preserve the per-instance working
directory, even after the first call to getcwd fails.
rdar://108213753
Differential Revision: https://reviews.llvm.org/D149173
value() has undesired exception checking semantics and calls
__throw_bad_optional_access in libc++. Moreover, the API is unavailable without
_LIBCPP_NO_EXCEPTIONS on older Mach-O platforms (see
_LIBCPP_AVAILABILITY_BAD_OPTIONAL_ACCESS).
This commit fixes LLVMAnalysis and its dependencies.
This change adds 'root-relative' option in vfsoverlay YAML file format
so the root patchs can be relative to the YAML file directory instead of
the current working directory.
Differential Revision: https://reviews.llvm.org/D137473
It is generally good practice, if you know how big the vector is going to be in the end, to reserve before continually calling "push_back" or "emplace_back"
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D139483
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
This patch implements symlinks for the in-memory VFS. Original author: @erik.pilkington.
Depends on D117648 & D117649.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D117650
If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.
Resolves rdar://90578880 and llvm-project#53306.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D123398
This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstances. Specifically,
since crash reproducers have `UseExternalNames` set to false, the
`File->getFileEntry().getDir()->getName()` call in `DoFrameworkLookup`
would use the *cached* directory name instead of the directory of the
looked-up file.
The plan is to re-commit this patch but to *add*
`ExposesExternalVFSPath` rather than replace `IsVFSMapped`.
Differential Revision: https://reviews.llvm.org/D123103
If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
This also renames `IsVFSMapped` to `ExposesExternalVFSPath` and only
sets it if `UseExternalName` is true. This flag then represents that the
`Status` has an external path that's different from its virtual path.
Right now the contained path is still the external path, but further PRs
will change this to *always* be the virtual path. Clients that need the
external can then request it specifically.
Note that even though `ExposesExternalVFSPath` isn't set for all
VFS-mapped paths, `IsVFSMapped` was only being used by a hack in
`FileManager` that was specific to module searching. In that case
`UseExternalNames` is always `true` and so that hack still applies.
Resolves rdar://90578880 and llvm-project#53306.
Differential Revision: https://reviews.llvm.org/D122549
For now most are implemented by printing out the name of the filesystem,
but this can be expanded in the future. Only `OverlayFileSystem` and
`RedirectingFileSystem` are properly implemented in this patch.
- `OverlayFileSystem`: Prints each filesystem in the order that any
operations are actually run on them. Optionally prints recursively.
- `RedirectingFileSystem`: Prints out all mappings, as well as the
`ExternalFS`. Most of this was already implemented other than the
handling for the `DirectoryRemap` case and to actually print out the
mapping.
Each FS should implement `printImpl` rather than `print`, where the
latter just fowards to the former. This is to avoid spreading the
default arguments through to the subclasses (where we may miss updating
in the future).
Differential Revision: https://reviews.llvm.org/D121421
The rest of LLVM uses `print` for the method taking the `raw_ostream`
and `dump` only for the method with no parameters. Use the same for
`RedirectingFileSystem`.
Differential Revision: https://reviews.llvm.org/D121494
Errors are generally checked in clients by comparing to the portable
error condition in `std::errc`, which will have the `generic_category`
(eg. `std::errc::no_such_file_or_directory`). While in practice these
are usually equivalent for the standard errno's, they are not in *all*
implementations. One such example is CentOS 7.
Differential Revision: https://reviews.llvm.org/D120299
Extend "fallthrough" to allow a third option: "fallback". Fallthrough
allows the original path to used if the redirected (or mapped) path
fails. Fallback is the reverse of this, ie. use the original path and
fallback to the mapped path otherwise.
While this result *can* be achieved today using multiple overlays, this
adds a much more intuitive option. As an example, take two directories
"A" and "B". We would like files from "A" to be used, unless they don't
exist, in which case the VFS should fallback to those in "B".
With the current fallthrough option this is possible by adding two
overlays: one mapping from A -> B and another mapping from B -> A. Since
the frontend *nests* the two RedirectingFileSystems, the result will
be that "A" is mapped to "B" and back to "A", unless it isn't in "A" in
which case it fallsthrough to "B" (or fails if it exists in neither).
Using "fallback" semantics allows a single overlay instead: one mapping
from "A" to "B" but only using that mapping if the operation in "A"
fails first.
"redirect-only" is used to represent the current "fallthrough: false"
case.
Differential Revision: https://reviews.llvm.org/D117937
The cleanup was manual, but assisted by "include-what-you-use". It consists in
1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.
Notable changes:
- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h
You may need to add some of these headers in your compilation units, if needs be.
As an hint to the impact of the cleanup, running
clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l
before: 8000919 lines
after: 7917500 lines
Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)
Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831
The creation of in-memory VFS nodes happens in a single function that deduces what kind of node to create from the arguments. This leads to complicated if-then-else logic that's difficult to cleanly extend.
This patch abstracts away in-memory node creation via a type-erased factory function that's passed instead.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D117648
This patch virtualizes the `getStatus` function on `InMemoryNode` in LLVM VFS. Currently, this is implemented via top-level function `getNodeStatus` that tries to cast `InMemoryNode *` into each subtype. Virtual functions seem to be the simpler solution here.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D117649
This diff adds support for relative roots to VFS overlays. The directory root
will be made absolute from the current working directory and will be used to
determine the path style to use. This supports the use of VFS overlays with
remote build systems that might use a different working directory for each
compilation.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D116174
The minimizing and caching filesystem used by the dependency scanner keeps minimized and original files in separate caches.
This setup is not well suited for dealing with files that are sometimes minimized and sometimes not. Such files are being stat-ed and read twice, which is wasteful and also means the two versions of the file can get "out of sync".
This patch squashes the two caches together. When a file is stat-ed or read, its original contents are populated. If a file needs to be minimized, we give the minimizer the already loaded contents instead of reading the file again.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D115346
This reverts commit f0cf544d6f6fe6cbca4c07772998272d6bb433d8.
Just a small change to fix:
```
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp: In static member function ‘static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)’:
/home/buildbot/as-builder-4/llvm-clang-x86_64-expensive-checks-ubuntu/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert ‘F’ from ‘std::unique_ptr<llvm::vfs::File>’ to ‘llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >’
return F;
^
```
Differential Revision: https://reviews.llvm.org/D113832
```
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp: In static member function 'static llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> > llvm::vfs::File::getWithPath(llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >, const llvm::Twine&)':
/work/omp-vega20-0/openmp-offload-amdgpu-runtime/llvm.src/llvm/lib/Support/VirtualFileSystem.cpp:2084:10: error: could not convert 'F' from 'std::unique_ptr<llvm::vfs::File>' to 'llvm::ErrorOr<std::unique_ptr<llvm::vfs::File> >'
return F;
^
```
This reverts commit c972175649f4bb50d40d911659a04d5620ce6fe0.
