Introducing `llvm::createStringErrorV` caused a `0.5%` compile-time
regression because it's an inline function in a core header. This moves
the API to a new header to prevent including this function in files that
don't need it.
Also includes the header in the source files that have been using
`createStringErrorV` (which currently is just LLDB).
This relands #165277 by reverting #169397.
This also relands the corresponding Bazel port by reverting #169410.
The original revert was due to a report of a broken build, which was
later resolved by fully clearing the build directory.
This removes the dependency on clangDriver from clangFrontend and
flangFrontend.
This refactoring is part of a broader effort to support driver-managed
builds for compilations using C++ named modules and/or Clang modules.
It is required for linking the dependency scanning tooling against the
driver without introducing cyclic dependencies, which would otherwise
cause build failures when dynamic linking is enabled.
In particular, clangFrontend must no longer depend on clangDriver
for this to be possible.
This change was discussed in the following RFC:
https://discourse.llvm.org/t/rfc-new-clangoptions-library-remove-dependency-on-clangdriver-from-clangfrontend-and-flangfrontend/88773
This got exposed by `09262656f32ab3f2e1d82e5342ba37eecac52522`.
The underlying stream of `m_os` is referenced by the `TextDiagnostic`
member of `TextDiagnosticPrinter`. It got turned into a
`llvm::formatted_raw_ostream` in the commit above. When
`~TextDiagnosticPrinter` (and thus `~TextDiagnostic`) is invoked, we now
call `~formatted_raw_ostream`, which tries to access the underlying
stream. But `m_os` was already deleted because it is earlier in the
order of destruction in `TextDiagnosticPrinter`. Move the `m_os` member
before the `TextDiagnosticPrinter` to avoid a use-after-free.
Drive-by:
* Also move the `m_output` member which the `m_os` holds a reference to.
The fact it's a reference indicates the expectation is most likely that
the string outlives the stream.
The ASAN macOS bot is currently failing with this:
```
08:15:39 =================================================================
08:15:39 ==61103==ERROR: AddressSanitizer: heap-use-after-free on address 0x60600012cf40 at pc 0x00012140d304 bp 0x00016eecc850 sp 0x00016eecc848
08:15:39 READ of size 8 at 0x60600012cf40 thread T0
08:15:39 #0 0x00012140d300 in llvm::formatted_raw_ostream::releaseStream() FormattedStream.h:205
08:15:39 #1 0x00012140d3a4 in llvm::formatted_raw_ostream::~formatted_raw_ostream() FormattedStream.h:145
08:15:39 #2 0x00012604abf8 in clang::TextDiagnostic::~TextDiagnostic() TextDiagnostic.cpp:721
08:15:39 #3 0x00012605dc80 in clang::TextDiagnosticPrinter::~TextDiagnosticPrinter() TextDiagnosticPrinter.cpp:30
08:15:39 #4 0x00012605dd5c in clang::TextDiagnosticPrinter::~TextDiagnosticPrinter() TextDiagnosticPrinter.cpp:27
08:15:39 #5 0x0001231fb210 in (anonymous namespace)::StoringDiagnosticConsumer::~StoringDiagnosticConsumer() ClangModulesDeclVendor.cpp:47
08:15:39 #6 0x0001231fb3bc in (anonymous namespace)::StoringDiagnosticConsumer::~StoringDiagnosticConsumer() ClangModulesDeclVendor.cpp:47
08:15:39 #7 0x000129aa9d70 in clang::DiagnosticsEngine::~DiagnosticsEngine() Diagnostic.cpp:91
08:15:39 #8 0x0001230436b8 in llvm::RefCountedBase<clang::DiagnosticsEngine>::Release() const IntrusiveRefCntPtr.h:103
08:15:39 #9 0x0001231fe6c8 in (anonymous namespace)::ClangModulesDeclVendorImpl::~ClangModulesDeclVendorImpl() ClangModulesDeclVendor.cpp:93
08:15:39 #10 0x0001231fe858 in (anonymous namespace)::ClangModulesDeclVendorImpl::~ClangModulesDeclVendorImpl() ClangModulesDeclVendor.cpp:93
...
08:15:39
08:15:39 0x60600012cf40 is located 32 bytes inside of 56-byte region [0x60600012cf20,0x60600012cf58)
08:15:39 freed by thread T0 here:
08:15:39 #0 0x0001018abb88 in _ZdlPv+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4bb88)
08:15:39 #1 0x0001231fb1c0 in (anonymous namespace)::StoringDiagnosticConsumer::~StoringDiagnosticConsumer() ClangModulesDeclVendor.cpp:47
08:15:39 #2 0x0001231fb3bc in (anonymous namespace)::StoringDiagnosticConsumer::~StoringDiagnosticConsumer() ClangModulesDeclVendor.cpp:47
08:15:39 #3 0x000129aa9d70 in clang::DiagnosticsEngine::~DiagnosticsEngine() Diagnostic.cpp:91
08:15:39 #4 0x0001230436b8 in llvm::RefCountedBase<clang::DiagnosticsEngine>::Release() const IntrusiveRefCntPtr.h:103
08:15:39 #5 0x0001231fe6c8 in (anonymous namespace)::ClangModulesDeclVendorImpl::~ClangModulesDeclVendorImpl() ClangModulesDeclVendor.cpp:93
08:15:39 #6 0x0001231fe858 in (anonymous namespace)::ClangModulesDeclVendorImpl::~ClangModulesDeclVendorImpl() ClangModulesDeclVendor.cpp:93
...
08:15:39
08:15:39 previously allocated by thread T0 here:
08:15:39 #0 0x0001018ab760 in _Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4b760)
08:15:39 #1 0x0001231f8dec in lldb_private::ClangModulesDeclVendor::Create(lldb_private::Target&) ClangModulesDeclVendor.cpp:732
08:15:39 #2 0x00012320af58 in lldb_private::ClangPersistentVariables::GetClangModulesDeclVendor() ClangPersistentVariables.cpp:124
08:15:39 #3 0x0001232111f0 in lldb_private::ClangUserExpression::PrepareForParsing(lldb_private::DiagnosticManager&, lldb_private::ExecutionContext&, bool) ClangUserExpression.cpp:536
08:15:39 #4 0x000123213790 in lldb_private::ClangUserExpression::Parse(lldb_private::DiagnosticManager&, lldb_private::ExecutionContext&, lldb_private::ExecutionPolicy, bool, bool) ClangUserExpression.cpp:647
08:15:39 #5 0x00012032b258 in lldb_private::UserExpression::Evaluate(lldb_private::ExecutionContext&, lldb_private::EvaluateExpressionOptions const&, llvm::StringRef, llvm::StringRef, std::__1::shared_ptr<lldb_private::ValueObject>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>*, lldb_private::ValueObject*) UserExpression.cpp:280
08:15:39 #6 0x000120724010 in lldb_private::Target::EvaluateExpression(llvm::StringRef, lldb_private::ExecutionContextScope*, std::__1::shared_ptr<lldb_private::ValueObject>&, lldb_private::EvaluateExpressionOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>*, lldb_private::ValueObject*) Target.cpp:2905
08:15:39 #7 0x00011fc7bde0 in lldb::SBTarget::EvaluateExpression(char const*, lldb::SBExpressionOptions const&) SBTarget.cpp:2305
08:15:39 ==61103==ABORTING
...
```
Depends on:
* https://github.com/llvm/llvm-project/pull/166917
When loading all Clang modules for a CU, we stop on first error. This
means benign module loading errors may stop us from importing actually
useful modules. There's no good reason to bail on the first one. The
pathological case would be if we try to load a large number of Clang
modules
but all fail to load for the same reason. That could happen, but in
practice I've always seen only a handful of modules failing to load out
of a large number. Particularly system modules are useful and usually
don't fail to load. Whereas project-specific Clang modules are more
likely to fail because the build system moves the modulemap/sources
around.
This patch accumulates all module loading errors and doesn't stop when
an error is encountered.
Instead of propagating the errors as a `bool`+`Stream` we change the
`ClangModulesDeclVendor` module loading APIs to use `llvm::Error`. We
also reword some of the diagnostics (notably removing the hardcoded
`error:` prefix). A follow-up patch will further make the module loading
errors less noisy.
See the new tests for what the errors look like.
rdar://164002569
The `ClangDeclVendor` used to contain more Clang-specific code than it
does nowadays. But at this point, all it does is wrap the
`DeclVendor::FindDecls` call and copy the resulting decls into
`std::vector<clang::NamedDecl*>`. I.e., it converts the generic
`CompilerDecl`s to `clang::NamedDecl*`s.
In my opinion at this point it doesn't do enough to justify making it
part of the `DeclVendor` hierarchy.
This patch removes the `ClangDeclVendor` and instead does the conversion
at callsite.
This PR is a part of the effort to make the VFS used in the compiler
more explicit and consistent.
Instead of creating the VFS deep within the compiler (in
`CompilerInstance::createFileManager()`), clients are now required to
explicitly call `CompilerInstance::createVirtualFileSystem()` and
provide the base VFS from the outside.
This PR also helps in breaking up the dependency cycle where creating a
properly configured `DiagnosticsEngine` requires a properly configured
VFS, but creating properly configuring a VFS requires the
`DiagnosticsEngine`.
Both `CompilerInstance::create{FileManager,Diagnostics}()` now just use
the VFS already in `CompilerInstance` instead of taking one as a
parameter, making the VFS consistent across the instance sub-object.
Handles clang::DiagnosticsEngine and clang::DiagnosticIDs.
For DiagnosticIDs, this mostly migrates from `new DiagnosticIDs` to
convenience method `DiagnosticIDs::create()`.
Part of cleanup https://github.com/llvm/llvm-project/issues/151026
This patch ensures we can find decls in submodules during expression
evaluation. Previously, submodules would have all their decls marked as
`Hidden`. When Clang asked LLDB for decls, it would see them in the
submodule but `clang::Sema` would reject them because they weren't
`Visible` (specifically, `getAcceptableDecl` would fail during
`CppNameLookup`). Here we just mark the submodule as visible to work
around this problem.
This reverts commit 57f3151a3144259f4e830fc43a1424e4c1f15985.
LLDB was hitting an assert when compiling the `std` module. The `std`
module was being pulled in because we use `#import <cstdio>` in the test
to set a breakpoint on `puts`. That's redundant and to work around the
crash we just remove that include. The underlying issue of compiling the
`std` module still exists and I'll investigate that separately. The
reason it started failing after the `ClangModulesDeclVendor` patch is that we would previously just fail to load the modulemap (and thus not load any of the modules). Now we do load the modulemap (and modules) when we prepare for parsing the expression.
The TestTemplateWithSameArg.py API test is crashing on macOS with
this change. Michael is going to look into it; reverting for now
to unblock the CI bots.
This reverts commit 052c70451afb7323ef72f321f3b0b5abb024b302.
`parseAndLoadModuleMapFile` returns `true` on error. This seems to have
always been an issue? This is now preventing me from fixing a different
modules related issue. So this patch checks the return value correctly.
This reverts commit e2a885537f11f8d9ced1c80c2c90069ab5adeb1d. Build failures were fixed right away and reverting the original commit without the fixes breaks the build again.
The `DiagnosticOptions` class is currently intrusively
reference-counted, which makes reasoning about its lifetime very
difficult in some cases. For example, `CompilerInvocation` owns the
`DiagnosticOptions` instance (wrapped in `llvm::IntrusiveRefCntPtr`) and
only exposes an accessor returning `DiagnosticOptions &`. One would
think this gives `CompilerInvocation` exclusive ownership of the object,
but that's not the case:
```c++
void shareOwnership(CompilerInvocation &CI) {
llvm::IntrusiveRefCntPtr<DiagnosticOptions> CoOwner = &CI.getDiagnosticOptions();
// ...
}
```
This is a perfectly valid pattern that is being actually used in the
codebase.
I would like to ensure the ownership of `DiagnosticOptions` by
`CompilerInvocation` is guaranteed to be exclusive. This can be
leveraged for a copy-on-write optimization later on. This PR changes
usages of `DiagnosticOptions` across `clang`, `clang-tools-extra` and
`lldb` to not be intrusively reference-counted.
Instead of eagerly populating the `clang::ModuleMap` when looking up a
module by name, this patch changes `HeaderSearch` to only load the
modules that are actually used.
This introduces `ModuleMap::findOrLoadModule` which will load modules
from parsed but not loaded module maps. This cannot be used anywhere
that the module loading code calls into as it can create infinite
recursion.
This currently just reparses module maps when looking up a module by
header. This is fine as redeclarations are allowed from the same file,
but future patches will also make looking up a module by header lazy.
This patch changes the shadow.m test to use explicitly built modules and
`#import`. This test and the shadow feature are very brittle and do not
work in general. The test relied on pcm files being left behind by prior
failing clang invocations that were then reused by the last invocation.
If you clean the cache then the last invocation will always fail. This
is because the input module map and the `-fmodule-map-file=` module map
are parsed in the same module scope, and `-fmodule-map-file=` is
forwarded to implicit module builds. That means you are guaranteed to
hit a module redeclaration error if the TU actually imports the module
it is trying to shadow.
This patch changes when we load A2's module map to after the `A` module
has been loaded, which sets the `IsFromModuleFile` bit on `A`. This
means that A2's `A` is skipped entirely instead of creating a shadow
module, and we get textual inclusion. It is possible to construct a case
where this would happen before this patch too.
An upcoming patch in this series will rework shadowing to work in the
general case, but that's only possible once header -> module lookup is
lazy too.
This PR makes it so that `CompilerInvocation` needs to be provided to
`CompilerInstance` on construction. There are a couple of benefits in my
view:
* Making it impossible to mis-use some `CompilerInstance` APIs. For
example there are cases, where `createDiagnostics()` was called before
`setInvocation()`, causing the `DiagnosticEngine` to use the
default-constructed `DiagnosticOptions` instead of the intended ones.
* This shrinks `CompilerInstance`'s state space.
* This makes it possible to access **the** invocation in
`CompilerInstance`'s constructor (to be used in a follow-up).
This PR hides the reference-counted pointer that holds `TargetOptions`
from the public API of `CompilerInvocation`. This gives
`CompilerInvocation` an exclusive control over the lifetime of this
member, which will eventually be leveraged to implement a copy-on-write
behavior.
There are two clients that currently share ownership of that pointer:
* `TargetInfo` - This was refactored to hold a non-owning reference to
`TargetOptions`. The options object is typically owned by the
`CompilerInvocation` or by the new `CompilerInstance::AuxTargetOpts` for
the auxiliary target. This needed a bit of care in `ASTUnit::Parse()` to
keep the `CompilerInvocation` alive.
* `clangd::PreambleData` - This was refactored to exclusively own the
`TargetOptions` that get moved out of the `CompilerInvocation`.
This PR reland https://github.com/llvm/llvm-project/pull/135808, fixed
some missed changes in LLDB.
I found this issue when I working on
https://github.com/llvm/llvm-project/pull/107168.
Currently we have many similiar data structures like:
- std::pair<IdentifierInfo *, SourceLocation>.
- Element type of ModuleIdPath.
- IdentifierLocPair.
- IdentifierLoc.
This PR unify these data structures to IdentifierLoc, moved
IdentifierLoc definition to SourceLocation.h, and deleted other similer
data structures.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
As specified in the docs,
1) raw_string_ostream is always unbuffered and
2) the underlying buffer may be used directly
( 65b13610a5226b84889b923bae884ba395ad084d for further reference )
* Don't call raw_string_ostream::flush(), which is essentially a no-op.
* Avoid unneeded calls to raw_string_ostream::str(), to avoid excess
indirection.
Following the work done by @jdevlieghere in D143690, this changes how Clang module build
events are emitted.
Instead of one Progress event per module being built, a single Progress event is used to
encompass all modules, and each module build is sent as an `Increment` update.
Differential Revision: https://reviews.llvm.org/D147248
Update the Clang diagnostic consumer (in ClangModulesDeclVendor) to report
progress on Clang module builds, as both progress events and expression logs.
Module build remarks are enabled by with clang's `-Rmodule-build` flag.
With this change, command line users of lldb will see progress events showing
which modules are being built, and - by how long they stay on screen - how much
time it takes to build them. IDEs that show progress events can show these
updates if desired.
This does not show module-import remarks, although that may be added as a
future change.
Differential Revision: https://reviews.llvm.org/D140056
D127284 introduced a new language option which is not benign from modules
perspective. Before this patch lldb would set up the compiler invocation and
later enable incremental processing. Post-D127284 this does not work because
the option causes a module hash mismatch for implicit modules.
In addition, D127284 enables parsing statements on the global scope if
incremental processing is on and thus `syntax_error_for_lldb_to_find` was
rightfully not recognized as a declaration and is considered a statement
which produces a slightly different diagnostic.
Thanks to Michael Buch for the help in understanding this issue. This patch
should appease the lldb bots.
More discussion available at: https://reviews.llvm.org/D127284
When a process gets restarted TypeSystem objects associated with it
may get deleted, and any CompilerType objects holding on to a
reference to that type system are a use-after-free in waiting. Because
of the SBAPI, we don't have tight control over where CompilerTypes go
and when they are used. This is particularly a problem in the Swift
plugin, where the scratch TypeSystem can be restarted while the
process is still running. The Swift plugin has a lock to prevent
abuse, but where there's a lock there can be bugs.
This patch changes CompilerType to store a std::weak_ptr<TypeSystem>.
Most of the std::weak_ptr<TypeSystem>* uglyness is hidden by
introducing a wrapper class CompilerType::WrappedTypeSystem that has a
dyn_cast_or_null() method. The only sites that need to know about the
weak pointer implementation detail are the ones that deal with
creating TypeSystems.
rdar://101505232
Differential Revision: https://reviews.llvm.org/D136650
In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.
We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).
Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:
import module A;
int some_decl();
import module B;
where being in a module purview is insufficient to identify them.
Differential Revision: https://reviews.llvm.org/D118893
Recently we observed high memory pressure caused by clang during some parallel builds.
We discovered that we have several projects that have a large number of #define directives
in their TUs (on the order of millions), which caused huge memory consumption in clang due
to a lot of allocations for MacroInfo. We would like to reduce the memory overhead of
clang for a single #define to reduce the memory overhead for these files, to allow us to
reduce the memory pressure on the system during highly parallel builds. This change achieves
that by removing the SmallVector in MacroInfo and instead storing the tokens in an array
allocated using the bump pointer allocator, after all tokens are lexed.
The added unit test with 1000000 #define directives illustrates the problem. Prior to this
change, on arm64 macOS, clang's PP bump pointer allocator allocated 272007616 bytes, and
used roughly 272 bytes per #define. After this change, clang's PP bump pointer allocator
allocates 120002016 bytes, and uses only roughly 120 bytes per #define.
For an example test file that we have internally with 7.8 million #define directives, this
change produces the following improvement on arm64 macOS: Persistent allocation footprint for
this test case file as it's being compiled to LLVM IR went down 22% from 5.28 GB to 4.07 GB
and the total allocations went down 14% from 8.26 GB to 7.05 GB. Furthermore, this change
reduced the total number of allocations made by the system for this clang invocation from
1454853 to 133663, an order of magnitude improvement.
The recommit fixes the LLDB build failure.
Differential Revision: https://reviews.llvm.org/D117348
Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
infrastructure). This worked because Log.h included Logging.h, even
though it should.
After the recent refactor, it became impossible the two files include
each other in this direction (the opposite inclusion is needed), so this
patch removes the workaround that was put in place and cleans up all
files to include the right thing. It also renames the file to LLDBLog to
better reflect its purpose.
