The patch reapply https://github.com/llvm/llvm-project/pull/173130.
This patch implement the following papers:
[P1857R3 Modules Dependency Discovery](https://wg21.link/p1857r3).
[P3034R1 Module Declarations Shouldn’t be
Macros](https://wg21.link/P3034R1).
[CWG2947](https://cplusplus.github.io/CWG/issues/2947.html).
At the start of phase 4 an import or module token is treated as starting
a directive and are converted to their respective keywords iff:
- After skipping horizontal whitespace are
- at the start of a logical line, or
- preceded by an export at the start of the logical line.
- Are followed by an identifier pp token (before macro expansion), or
- <, ", or : (but not ::) pp tokens for import, or
- ; for module
Otherwise the token is treated as an identifier.
Additionally:
- The entire import or module directive (including the closing ;) must
be on a single logical line and for module must not come from an
#include.
- The expansion of macros must not result in an import or module
directive introducer that was not there prior to macro expansion.
- A module directive may only appear as the first preprocessing tokens
in a file (excluding the global module fragment.)
- Preprocessor conditionals shall not span a module declaration.
After this patch, we handle C++ module-import and module-declaration as
a real pp-directive in preprocessor. Additionally, we refactor module
name lexing, remove the complex state machine and read full module name
during module/import directive handling. Possibly we can introduce a
tok::annot_module_name token in the future, avoid duplicatly parsing
module name in both preprocessor and parser, but it's makes error
recovery much diffcult(eg. import a; import b; in same line).
This patch also introduce 2 new keyword `__preprocessed_module` and
`__preprocessed_import`. These 2 keyword was generated during `-E` mode.
This is useful to avoid confusion with `module` and `import` keyword in
preprocessed output:
```cpp
export module m;
struct import {};
#define EMPTY
EMPTY import foo;
```
Fixes https://github.com/llvm/llvm-project/issues/54047
The previous patch has an use-after-free issue in
Lexer::LexTokenInternal function. Since C++20, the `export`, `import`
and `module` identifiers may be an introducer of a C++ module
declaration/importing directive, and the directive will handled in
`LexIdentifierContinue`. Unfortunately, the EOF may be encountered in
`LexIdentifierContinue` and `CurLexer` might be destructed in
`HandleEndOfFile`, If the code after `LexIdentifierContinue` try to
access `LangOps` or other class members in this Lexer, it will hit
undefined behavior.
This patch also fix the use-after-free issue in Lexer by introduce a
mechanism to delay the destruction of `CurLexer` in `Preprocessor`
class.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR implement the following papers:
[P1857R3 Modules Dependency Discovery](https://wg21.link/p1857r3).
[P3034R1 Module Declarations Shouldn’t be
Macros](https://wg21.link/P3034R1).
[CWG2947](https://cplusplus.github.io/CWG/issues/2947.html).
At the start of phase 4 an import or module token is treated as starting
a directive and are converted to their respective keywords iff:
- After skipping horizontal whitespace are
- at the start of a logical line, or
- preceded by an export at the start of the logical line.
- Are followed by an identifier pp token (before macro expansion), or
- <, ", or : (but not ::) pp tokens for import, or
- ; for module
Otherwise the token is treated as an identifier.
Additionally:
- The entire import or module directive (including the closing ;) must
be on a single logical line and for module must not come from an
#include.
- The expansion of macros must not result in an import or module
directive introducer that was not there prior to macro expansion.
- A module directive may only appear as the first preprocessing tokens
in a file (excluding the global module fragment.)
- Preprocessor conditionals shall not span a module declaration.
After this patch, we handle C++ module-import and module-declaration as
a real pp-directive in preprocessor. Additionally, we refactor module
name lexing, remove the complex state machine and read full module name
during module/import directive handling. Possibly we can introduce a
tok::annot_module_name token in the future, avoid duplicatly parsing
module name in both preprocessor and parser, but it's makes error
recovery much diffcult(eg. import a; import b; in same line).
This patch also introduce 2 new keyword `__preprocessed_module` and
`__preprocessed_import`. These 2 keyword was generated during `-E` mode.
This is useful to avoid confusion with `module` and `import` keyword in
preprocessed output:
```cpp
export module m;
struct import {};
#define EMPTY
EMPTY import foo;
```
Fixes https://github.com/llvm/llvm-project/issues/54047
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Signed-off-by: Wang, Yihan <yronglin777@gmail.com>
Fixes#159768.
When building a named module interface with `-fmodules` enabled,
importing a Clang module from inside the global module fragment causes
Clang to crash only on assertion builds.
This fixes the assert and extends the test coverage.
This patch replaces SmallSet<T *, N> with SmallPtrSet<T *, N>. Note
that SmallSet.h "redirects" SmallSet to SmallPtrSet for pointer
element types:
template <typename PointeeType, unsigned N>
class SmallSet<PointeeType*, N> : public SmallPtrSet<PointeeType*, N>
{};
We only have 30 instances that rely on this "redirection", with about
half of them under clang/. Since the redirection doesn't improve
readability, this patch replaces SmallSet with SmallPtrSet for pointer
element types.
I'm planning to remove the redirection eventually.
Consider the following code:
```cpp
# 1 __FILE__ 1 3
export module a;
```
According to the wording in
[P1857R3](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1857r3.html):
```
A module directive may only appear as the first preprocessing tokens in a file (excluding the global module fragment.)
```
and the wording in
[[cpp.pre]](https://eel.is/c++draft/cpp.pre#nt:module-file)
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
`#` is the first pp-token in the translation unit, and it was rejected
by clang, but they really should be exempted from this rule. The goal is
to not allow any preprocessor conditionals or most state changes, but
these don't fit that.
State change would mean most semantically observable preprocessor state,
particularly anything that is order dependent. Global flags like being a
system header/module shouldn't matter.
We should exempt a brunch of directives, even though it violates the
current standard wording.
In this patch, we introduce a `TrivialDirectiveTracer` to trace the
**State change** that described above and propose to exempt the
following kind of directive: `#line`, GNU line marker, `#ident`,
`#pragma comment`, `#pragma mark`, `#pragma detect_mismatch`, `#pragma
clang __debug`, `#pragma message`, `#pragma GCC warning`, `#pragma GCC
error`, `#pragma gcc diagnostic`, `#pragma OPENCL EXTENSION`, `#pragma
warning`, `#pragma execution_character_set`, `#pragma clang
assume_nonnull` and builtin macro expansion.
Fixes https://github.com/llvm/llvm-project/issues/145274
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
It is better to use DynamicRecursiveASTVisitor than RecursiveASTVisitor
as it can reduce the generated size. And also avoid using a template
type to present callbacks to avoid generating more code too.
Tracked at https://github.com/llvm/llvm-project/issues/112294
This patch implements from [basic.link]p14 to [basic.link]p18 partially.
The explicitly missing parts are:
- Anything related to specializations.
- Decide if a pointer is associated with a TU-local value at compile
time.
- [basic.link]p15.1.2 to decide if a type is TU-local.
- Diagnose if TU-local functions from other TU are collected to the
overload set. See [basic.link]p19, the call to 'h(N::A{});' in
translation unit #2
There should be other implicitly missing parts as the wording uses
"names" briefly several times. But to implement this precisely, we have
to visit the whole AST, including Decls, Expression and Types, which may
be harder to implement and be more time-consuming for compilation time.
So I choose to implement the common parts.
It won't be too bad to miss some cases since we DIDN'T do any such
checks in the past 3 years. Any new check is an improvement. Given
modules have been basically available since clang15 without such checks,
it will be user unfriendly if we give a hard error now. And there are
a lot of cases which violating the rule actually just fine. So I decide
to emit it as warnings instead of hard errors.
Depends on [[clang][Preprocessor] Add peekNextPPToken, makes look ahead
next token without
side-effects](https://github.com/llvm/llvm-project/pull/143898).
This PR fix the performance regression that introduced in
https://github.com/llvm/llvm-project/pull/144233.
The original PR(https://github.com/llvm/llvm-project/pull/144233) handle
the first pp-token in the main source file in the macro
definition/expansion and `Lexer::Lex`, but the lexer is almost always on
the hot path, we may hit a performance regression. In this PR, we handle
the first pp-token in `Preprocessor::EnterMainSourceFile`.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR is 2nd part of
[P1857R3](https://github.com/llvm/llvm-project/pull/107168)
implementation, and mainly implement the restriction `A module directive
may only appear as the first preprocessing tokens in a file (excluding
the global module fragment.)`:
[cpp.pre](https://eel.is/c++draft/cpp.pre):
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
We also refine tests use `split-file` instead of conditional macro.
Signed-off-by: yronglin <yronglin777@gmail.com>
Instead of eagerly populating the `clang::ModuleMap` when looking up a
module by name, this patch changes `HeaderSearch` to only load the
modules that are actually used.
This introduces `ModuleMap::findOrLoadModule` which will load modules
from parsed but not loaded module maps. This cannot be used anywhere
that the module loading code calls into as it can create infinite
recursion.
This currently just reparses module maps when looking up a module by
header. This is fine as redeclarations are allowed from the same file,
but future patches will also make looking up a module by header lazy.
This patch changes the shadow.m test to use explicitly built modules and
`#import`. This test and the shadow feature are very brittle and do not
work in general. The test relied on pcm files being left behind by prior
failing clang invocations that were then reused by the last invocation.
If you clean the cache then the last invocation will always fail. This
is because the input module map and the `-fmodule-map-file=` module map
are parsed in the same module scope, and `-fmodule-map-file=` is
forwarded to implicit module builds. That means you are guaranteed to
hit a module redeclaration error if the TU actually imports the module
it is trying to shadow.
This patch changes when we load A2's module map to after the `A` module
has been loaded, which sets the `IsFromModuleFile` bit on `A`. This
means that A2's `A` is skipped entirely instead of creating a shadow
module, and we get textual inclusion. It is possible to construct a case
where this would happen before this patch too.
An upcoming patch in this series will rework shadowing to work in the
general case, but that's only possible once header -> module lookup is
lazy too.
This PR reland https://github.com/llvm/llvm-project/pull/135808, fixed
some missed changes in LLDB.
I found this issue when I working on
https://github.com/llvm/llvm-project/pull/107168.
Currently we have many similiar data structures like:
- std::pair<IdentifierInfo *, SourceLocation>.
- Element type of ModuleIdPath.
- IdentifierLocPair.
- IdentifierLoc.
This PR unify these data structures to IdentifierLoc, moved
IdentifierLoc definition to SourceLocation.h, and deleted other similer
data structures.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Reverts llvm/llvm-project#135808
Example from the LLDB macOS CI:
https://green.lab.llvm.org/job/llvm.org/view/LLDB/job/as-lldb-cmake/24084/execution/node/54/log/?consoleFull
```
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/lldb/source/Plugins/ExpressionParser/Clang/ClangModulesDeclVendor.cpp:360:49: error: no viable conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'clang::ModuleIdPath' (aka 'ArrayRef<IdentifierLoc>')
clang::Module *top_level_module = DoGetModule(clang_path.front(), false);
^~~~~~~~~~~~~~~~~~
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:41:40: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'const llvm::ArrayRef<clang::IdentifierLoc> &' for 1st argument
class LLVM_GSL_POINTER [[nodiscard]] ArrayRef {
^
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:41:40: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'llvm::ArrayRef<clang::IdentifierLoc> &&' for 1st argument
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:70:18: note: candidate constructor not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'std::nullopt_t' for 1st argument
/*implicit*/ ArrayRef(std::nullopt_t) {}
```
I found this issue when I working on
https://github.com/llvm/llvm-project/pull/107168.
Currently we have many similiar data structures like:
- `std::pair<IdentifierInfo *, SourceLocation>`.
- Element type of `ModuleIdPath`.
- `IdentifierLocPair`.
- `IdentifierLoc`.
This PR unify these data structures to `IdentifierLoc`, moved
`IdentifierLoc` definition to SourceLocation.h, and deleted other
similer data structures.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Recently, there are multiple false positive issue reports about the
reachability of implementation partition units:
- https://github.com/llvm/llvm-project/issues/105882
- https://github.com/llvm/llvm-project/issues/101348
- https://lists.isocpp.org/core/2024/08/16232.php
And according to our use experience for modules, we find it is a pretty
good practice to not import implementation partition units in the
interface units. It can help developers to have a pretty good mental
model for when to use an implementation partition unit: that any unit in
the module but not in the module interfaces can be in the implementation
partition unit.
So I think it is good to add the diagnostics.
Implements `export` keyword in HLSL.
There are two ways the `export` keyword can be used:
1. On individual function declarations
```
export void f() {}
```
2. On a group of function declaration:
```
export {
void f1();
void f2() {}
}
```
Functions declared with the `export` keyword have external linkage. The
implementation does not include validation of when a function can or
cannot be exported, such as when it has resource argument or semantic
annotations. That will be covered by llvm/llvm-project#93330.
Currently all function declarations in global or named namespaces have
external linkage by default so there are no specific code changes
required right now to make sure exported function have external linkage
as well. That will change as part of llvm/llvm-project#92071. Any
additional changes to make sure exported functions still have external
linkage will be done as part of this work item.
Fixes#92812
This patch moves documentation of `Sema` functions from `.cpp` files to `Sema.h` when there was no documentation in the latter, or it can be trivially subsumed. More complicated cases when there's less trivial divergence between documentation attached to declaration and the one attached to implementation are left for a later PR that would require review.
It appears that doxygen can find the documentation for a function defined out-of-line even if it's attached to an implementation, and not declaration. But other tools, e.g. clangd, are not as powerful. So this patch significantly improves autocompletion experience for (at least) clangd-based IDEs.
This patch extracts the logci to decide how we decide the module units
belongs to the same module into a member function of ASTContext. This is
helpful to refactor the implementation in the future.
Reduced BMI
Mitigate https://github.com/llvm/llvm-project/issues/61447
The root cause of the above problem is that when we write a declaration,
we need to lookup all the redeclarations in the imported modules. Then
it will be pretty slow if there are too many redeclarations in different
modules. This patch doesn't solve the porblem.
What the patchs mitigated is, when we writing a named module, we shouldn't
write the declarations from GMF if it is unreferenced **in current
module unit**. The difference here is that, if the declaration is used
in the imported modules, we used to emit it as an update. But we
definitely want to avoid that after this patch.
For that reproducer in
https://github.com/llvm/llvm-project/issues/61447, it used to take 2.5s
to compile and now it only takes 0.49s to compile, which is a big win.
This patch tries to make the boundary of clang module and C++20 named
module more clear.
The changes included:
- Rename `TranslationUnitKind::TU_Module` to
`TranslationUnitKind::TU_ClangModule`.
- Rename `Sema::ActOnModuleInclude` to `Sema::ActOnAnnotModuleInclude`.
- Rename `ActOnModuleBegin` to `Sema::ActOnAnnotModuleBegin`.
- Rename `Sema::ActOnModuleEnd` to `Sema::ActOnAnnotModuleEnd`.
- Removes a warning if we're trying to compile a non-module unit as
C++20 module unit. This is not actually useful and makes (the future)
implementation unnecessarily complex.
This patch meant to be a NFC fix. But it shows that it fixed a bug
suprisingly that previously we would surppress the unused-value warning
in named modules. Because it shares the same logic with clang modules,
which has headers semantics. This shows the change is meaningful.
Close https://github.com/llvm/llvm-project/issues/84002
Per [module.import]p7:
> Additionally, when a module-import-declaration in a module unit of
> some module M imports another module unit U of M, it also imports all
> translation units imported by non-exported module-import-declarations
> in the module unit purview of U.
However, we only tried to implement it during the implicit import of
primary module interface for module implementation unit.
Also we didn't implement the last sentence from [module.import]p7
completely:
> These rules can in turn lead to the importation of yet more
> translation units.
This patch tries to care the both issues.
Previously we forbiden the users to import named modules from clang header
modules. However, due to an oversight, the @import form of Objective C
got involved. This is not want and we fix that in this patch.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
Close https://github.com/llvm/llvm-project/issues/71347
Previously I misread the concept of module purview. I thought if a
declaration attached to a unnamed module, it can't be part of the module
purview. But after the issue report, I recognized that module purview is
more of a concept about locations instead of semantics.
Concretely, the things in the language linkage after module declarations
can be exported.
This patch refactors `Module::isModulePurview()` and introduces some
possible code cleanups.
The 'ModuleInterface' in Sema::ModuleScope is confusing. It actually
means 'not implementation'. This patch removes that bit and extract the
information from the recorded clang::Module.
This patch introduces a new enumerator `Invalid = 0`, shifting other enumerators by +1. Contrary to how it might sound, this actually affirms status quo of how this enum is stored in `clang::Decl`:
```
/// If 0, we have not computed the linkage of this declaration.
/// Otherwise, it is the linkage + 1.
mutable unsigned CacheValidAndLinkage : 3;
```
This patch makes debuggers to not be mistaken about enumerator stored in this bit-field. It also converts `clang::Linkage` to a scoped enum.
This patch converts `LinkageSpecDecl::LanguageIDs` into scoped enum, and moves it to namespace scope, so that it can be forward-declared where required.
This reapplies ddbcc10b9e26b18f6a70e23d0611b9da75ffa52f, except for a tiny part that was reverted separately: 65331da0032ab4253a4bc0ddcb2da67664bd86a9. That will be reapplied later on, since it turned out to be more involved.
This commit is enabled by 5523fefb01c282c4cbcaf6314a9aaf658c6c145f and f0f548a65a215c450d956dbcedb03656449705b9, specifically the part that makes 'clang-tidy/checkers/misc/header-include-cycle.cpp' separator agnostic.
This commit replaces some calls to the deprecated `FileEntry::getName()` with `FileEntryRef::getName()` by swapping current usages of `SourceManager::getFileEntryForID()` with `SourceManager::getFileEntryRefForID()`. This lowers the number of usages of the deprecated `FileEntry::getName()` from 95 to 50.
Close https://github.com/llvm/llvm-project/issues/64755
This wouldn't affect the form @import as the test shows. The two
affected test case `diag-flags.cpp` and `diag-pragma.cpp` are old test
cases in 2017 and 2018, when we're not so clear about the direction of
modules. And the things that these 2 tests tested can be covered by
clang modules naturally. So I change the them into clang modules to
not block this patch.
In preparation for removing the `#include "llvm/ADT/StringExtras.h"`
from the header to source file of `llvm/Support/Error.h`, first add in
all the missing includes that were previously included transitively
through this header.
The following test fails to compile TU b.cpp because we are not making the transitively imported modules visible (per [module.import]/p7)
```
a.cppm:
export module a;
export int foo() {
return 42;
}
b.cppm:
export module b;
import a;
export int bar();
b.cpp:
module b;
int bar() {
return foo();
}
clang++ -c -std=c++2b -fmodule-output a.cppm
clang++ -c -std=c++2b -fmodule-output -fprebuilt-module-path=. b.cppm
clang++ -c -std=c++2b -fprebuilt-module-path=. b.cpp
b.cpp:4:12: error: declaration of 'foo' must be imported from module 'a' before it is required
return foo();
```
This is fixed by the following patch (which also addresses a FIXME in basic.def.odr/p6.cppm).
Differential Revision: https://reviews.llvm.org/D152746
It has been reported to that the current clang errors for, specifically,
static_assert in export contexts are a serious blocker to adoption of
modules in some cases.
There is also implementation divergence with GCC and MSVC allowing the
constructs mentioned below where clang currently rejects them with an
error.
The category of errors [for declarations in an exported context] is:
(unnamed, static_assert, empty and asm decls). These are now permitted
after P2615R1 which was approved by WG21 as a DR (and thus should be
applied to C++20 as well).
This patch removes these diagnostics and amends the testsuite accordingly.
Differential Revision: https://reviews.llvm.org/D152946
When modules are disabled, there's no loaded module for these import
decls to point at. This results in crashes when there are modulemap
files but no -fmodules flag (this configuration is used for layering
check violations).
This patch makes sure import declarations are introduced only when
modules are enabled, which makes this case similar to textual headers
(no import decls are created for #include of textual headers from a
modulemap).
Differential Revision: https://reviews.llvm.org/D152274
Decl::isInAnotherModuleUnit
Refactor `Sema::isModuleUnitOfCurrentTU` to `Decl::isInAnotherModuleUnit`
to make code simpler a little bit. Note that although this patch
introduces a FIXME, this is an existing issue and this patch just tries
to describe it explicitly.
Currently, the header units are rarely used and it is not well tested.
To avoid further misunderstandings, let's mark it as experimental and
emit a warning when users wants to import it.
This is discussed in modules developers meeting.
The original name "ASTContext::getNamedModuleForCodeGen" is not properly
reflecting the usage of the interface. This interface can be used to
judge the current module unit in both sema analysis and code generation.
So the original name was not so correct.
Decl::isInCurrentModuleUnit
Refactor `Sema::isModuleUnitOfCurrentTU` to `Decl::isInCurrentModuleUnit`
to make code simpler a little bit. Note that although this patch
introduces a FIXME, this is an existing issue and this patch just tries
to describe it explicitly.
