Alternatives to https://reviews.llvm.org/D153114.
Try to address https://github.com/clangd/clangd/issues/1293.
See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.
This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
to review and understand so that every one are in the same page:
> Don't attempt any cross-file or cross-version coordination: i.e. don't
> try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
> build the preamble.
This patch reflects the above opinions.
# Testing in real-world project
I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.
The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.
# What this patch does in detail
- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.
# What we need to do next
Let's split the TODOs into clang part and clangd part to make things
more clear.
The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.
TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.
And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).
These are still disabled by default, but will work in ObjC code if you
enable the `-import-insertions` flag.
Completion requires ASTSignals to be available; before ASTSignals are
available, we will always use #include. Once they are available, the
behavior varies as follows:
- For source files, use #import if the ObjC language flag is enabled
- For header files:
- If the ObjC language flag is disabled, use #include
- If the header file contains any #imports, use #import
- If the header file references any ObjC decls, use #import
- Otherwise, use #include
IncludeFixer support is similar, but it does not rely upon ASTSignals,
instead it does the above checks excluding the scan for ObjC symbols.
Differential Revision: https://reviews.llvm.org/D139458
I am working on support for forwarding parameter names in make_unique-like functions, first for inlay hints, later maybe for signature help.
For that to work generically, I'd like to parse all of these functions in the preamble. Not sure how this impacts performance on large codebases though.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D124688
This replaces the test removed in 51be7061d025139ba66869d5d99c7157a3ae9edd
It is more principled and tests more critical cases: a crash while parsing.
We need two pieces of plumbing:
- a way to re-enable the crashing #pragmas via a flag, to test parse crashes
- a bit of reshuffling around ASTWorker execution so that we set up the
crash handler in both sync/async modes.
Sync mode is useful for debugging, so I tested both.
Differential Revision: https://reviews.llvm.org/D112565
Background-indexing is fine, because it uses GlobalCompilationDatabase
to fetch the compile commands (hence uses CommandMangler), and creates
invocation through buildCompilerInvocation.
Depends on D106639.
Differential Revision: https://reviews.llvm.org/D106669
These can be invoked at different stages while building an AST to let
FeatureModules implement features on top of it. The patch also
introduces a sawDiagnostic hook, which can mutate the final clangd::Diag
while reading a clang::Diagnostic.
Differential Revision: https://reviews.llvm.org/D98499
These force a couple of flags or that are now on by default.
So the flags don't currently do anything unless the compile command has
-fno-recovery-ast explicitly.
(For turning recovery *off* for debugging we can inject the flag with config)
This leaves the command-line flags around with no effect, I'm planning to add
a "retired flag" mechanism shortly in a separate patch.
Differential Revision: https://reviews.llvm.org/D94724
Added some new ClangTidyOptionsProvider like classes designed for clangd work flow.
These providers are designed to source the options on the worker thread but in a thread safe manner.
This is done through making the options getter take a pointer to the filesystem used by the worker thread which natuarally is from a ThreadsafeFS.
Internal caching in the providers is also guarded.
The providers don't inherit from `ClangTidyOptionsProvider` instead they share a base class which is able to create a provider for the `ClangTidyContext` using a specific FileSystem.
This approach means one provider can be used for multiple contexts even though `ClangTidyContext` owns its provider.
Depends on D90531
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D91029
Summary:
Clangd uses FSProvider to get threadsafe views into file systems. This
patch changes naming to make that more explicit.
Depends on D81920
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81998
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
Summary:
- This option forces a preamble rebuild to handle the odd case
of a missing header file being added
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73916
Summary:
Those errors are exposed at the first character of a file,
for a lack of a better place.
Previously, all errors were stored inside the AST and report
accordingly. However, errors in command-line argument parsing could
result in failure to produce the AST, so we need an alternative ways to
report those errors.
We take the following approach in this patch:
- buildCompilerInvocation() now requires an explicit DiagnosticConsumer.
- TUScheduler and TestTU now collect the diagnostics produced when
parsing command line arguments.
If pasing of the AST failed, diagnostics are reported via a new
ParsingCallbacks::onFailedAST method.
If parsing of the AST succeeded, any errors produced during
command-line parsing are stored alongside the AST inside the
ParsedAST instance and reported as previously by calling the
ParsingCallbacks::onMainAST method;
- The client code that uses ClangdServer's DiagnosticConsumer
does not need to change, it will receive new diagnostics in the
onDiagnosticsReady() callback
Errors produced when parsing command-line arguments are collected using
the same StoreDiags class that is used to collect all other errors. They
are recognized by their location being invalid. IIUC, the location is
invalid as there is no source manager at this point, it is created at a
later stage.
Although technically we might also get diagnostics that mention the
command-line arguments FileID with after the source manager was created
(and they have valid source locations), we choose to not handle those
and they are dropped as not coming from the main file. AFAICT, those
diagnostics should always be notes, therefore it's safe to drop them
without loosing too much information.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: nridge, javed.absar, MaskRay, jkorous, arphaman, cfe-commits, gribozavr
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66759
llvm-svn: 370177
Summary:
This makes addDocument non-blocking and would also allow code completion
(in fallback mode) to run when worker waits for the compile command.
Reviewers: sammccall, ilya-biryukov
Reviewed By: ilya-biryukov
Subscribers: javed.absar, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D60607
llvm-svn: 358400
Summary:
This enables clangd to intercept compiler diagnostics and attach fixes (e.g. by
querying index). This patch adds missing includes for incomplete types e.g.
member access into class with only forward declaration. This would allow adding
missing includes for user-typed symbol names that are missing declarations
(e.g. typos) in the future.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D56903
llvm-svn: 352361
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
This patch moves the virtual file system form clang to llvm so it can be
used by more projects.
Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.
RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html
Differential revision: https://reviews.llvm.org/D52783
llvm-svn: 344140
Summary:
To aid debugging failures and crashes.
Only part of ignored diagnostics was logged before, now we log all of
them.
Reviewers: ioeric, hokein, sammccall
Reviewed By: hokein
Subscribers: klimek, jkorous-apple, cfe-commits
Differential Revision: https://reviews.llvm.org/D43123
llvm-svn: 324888
Summary:
This improves performance of code completion, because we avoid stating
the files from the preamble and never attempt to parse the files
without using the preamble if it's provided.
However, the change comes at a cost of sometimes providing incorrect
results when doing code completion after making actually considerable
changes to the files used in the preamble or the preamble itself.
Eventually the preamble will get rebuilt and code completion will
be providing the correct results.
Reviewers: bkramer, sammccall, jkorous-apple
Reviewed By: sammccall
Subscribers: klimek, cfe-commits
Differential Revision: https://reviews.llvm.org/D41991
llvm-svn: 322854
Summary: Shared details of ClangdUnit and CodeComplete moved to a new Compiler file.
Reviewers: ilya-biryukov
Subscribers: klimek, mgorny, cfe-commits
Differential Revision: https://reviews.llvm.org/D40719
llvm-svn: 319655
