Consider the following code:
```cpp
# 1 __FILE__ 1 3
export module a;
```
According to the wording in
[P1857R3](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1857r3.html):
```
A module directive may only appear as the first preprocessing tokens in a file (excluding the global module fragment.)
```
and the wording in
[[cpp.pre]](https://eel.is/c++draft/cpp.pre#nt:module-file)
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
`#` is the first pp-token in the translation unit, and it was rejected
by clang, but they really should be exempted from this rule. The goal is
to not allow any preprocessor conditionals or most state changes, but
these don't fit that.
State change would mean most semantically observable preprocessor state,
particularly anything that is order dependent. Global flags like being a
system header/module shouldn't matter.
We should exempt a brunch of directives, even though it violates the
current standard wording.
In this patch, we introduce a `TrivialDirectiveTracer` to trace the
**State change** that described above and propose to exempt the
following kind of directive: `#line`, GNU line marker, `#ident`,
`#pragma comment`, `#pragma mark`, `#pragma detect_mismatch`, `#pragma
clang __debug`, `#pragma message`, `#pragma GCC warning`, `#pragma GCC
error`, `#pragma gcc diagnostic`, `#pragma OPENCL EXTENSION`, `#pragma
warning`, `#pragma execution_character_set`, `#pragma clang
assume_nonnull` and builtin macro expansion.
Fixes https://github.com/llvm/llvm-project/issues/145274
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Previously, the newline after a module directive was not properly
captured and printed by `clang::printDependencyDirectivesAsSource`.
According to P1857R3, each directive must, after skipping horizontal
whitespace, appear at the start of a logical line. Because the newline
after module directives was missing, this invalidated the following
line.
This fixes tests that were previously in violation of P1857R3,
including for Objective-C directives, which should also comply with
P1857R3.
This also ensures that the global module fragment `module;` is captured
by the dependency directives scanner.
Introduce a type alias for the commonly used `std::pair<FileID,
unsigned>` to improve code readability, and make it easier for future
updates (64-bit source locations).
Depends on [[clang][Preprocessor] Add peekNextPPToken, makes look ahead
next token without
side-effects](https://github.com/llvm/llvm-project/pull/143898).
This PR fix the performance regression that introduced in
https://github.com/llvm/llvm-project/pull/144233.
The original PR(https://github.com/llvm/llvm-project/pull/144233) handle
the first pp-token in the main source file in the macro
definition/expansion and `Lexer::Lex`, but the lexer is almost always on
the hot path, we may hit a performance regression. In this PR, we handle
the first pp-token in `Preprocessor::EnterMainSourceFile`.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR follow the
suggestion(https://github.com/llvm/llvm-project/pull/143898#discussion_r2164253141)
to refine the implementation of `Preprocessor::isNextPPToken`, also use
C++ fold expression to refine `Token::isOneOf`. We don't need `bool
isOneOf(tok::TokenKind K1, tok::TokenKind K2) const` anymore.
In order to reduce the impact, specificed `TokenKind` is still passed to
`Token::isOneOf` and `Preprocessor::isNextPPTokenOneOf` as function
parameters.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR introduce a new function `peekNextPPToken`. It's an extension of
`isNextPPTokenLParen` and can makes look ahead one token in preprocessor
without side-effects.
It's also the 1st part of
https://github.com/llvm/llvm-project/pull/107168 and it was used to look
ahead next token then determine whether current lexing pp directive is
one of pp-import or pp-module directive.
At the start of phase 4 an import or module token is treated as starting
a directive and are converted to their respective keywords iff:
- After skipping horizontal whitespace are
- at the start of a logical line, or
- preceded by an export at the start of the logical line.
- Are followed by an identifier pp token (before macro expansion), or
- <, ", or : (but not ::) pp tokens for import, or
- ; for module
Otherwise the token is treated as an identifier.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR is 2nd part of
[P1857R3](https://github.com/llvm/llvm-project/pull/107168)
implementation, and mainly implement the restriction `A module directive
may only appear as the first preprocessing tokens in a file (excluding
the global module fragment.)`:
[cpp.pre](https://eel.is/c++draft/cpp.pre):
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
We also refine tests use `split-file` instead of conditional macro.
Signed-off-by: yronglin <yronglin777@gmail.com>
We can simplify the code with *Map::try_emplace where we need
default-constructed values while avoding calling constructors when
keys are already present.
Static analysis flagged the unconditional access of getExternalSource().
We don't initialize ExternalSource during construction but via
setExternalSource(). If this is not set it will violate the invariant
covered by the assert.
This adds a new diagnostic group, -Wc++-keyword, which is off by default
and grouped under -Wc++-compat. The diagnostic catches use of C++
keywords in C code.
This change additionally fixes an issue with -Wreserved-identifier not
diagnosing use of reserved identifiers in function parameter lists in a
function declaration which is not a definition.
Fixes https://github.com/llvm/llvm-project/issues/21898
This PR reland https://github.com/llvm/llvm-project/pull/135808, fixed
some missed changes in LLDB.
I found this issue when I working on
https://github.com/llvm/llvm-project/pull/107168.
Currently we have many similiar data structures like:
- std::pair<IdentifierInfo *, SourceLocation>.
- Element type of ModuleIdPath.
- IdentifierLocPair.
- IdentifierLoc.
This PR unify these data structures to IdentifierLoc, moved
IdentifierLoc definition to SourceLocation.h, and deleted other similer
data structures.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Reverts llvm/llvm-project#135808
Example from the LLDB macOS CI:
https://green.lab.llvm.org/job/llvm.org/view/LLDB/job/as-lldb-cmake/24084/execution/node/54/log/?consoleFull
```
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/lldb/source/Plugins/ExpressionParser/Clang/ClangModulesDeclVendor.cpp:360:49: error: no viable conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'clang::ModuleIdPath' (aka 'ArrayRef<IdentifierLoc>')
clang::Module *top_level_module = DoGetModule(clang_path.front(), false);
^~~~~~~~~~~~~~~~~~
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:41:40: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'const llvm::ArrayRef<clang::IdentifierLoc> &' for 1st argument
class LLVM_GSL_POINTER [[nodiscard]] ArrayRef {
^
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:41:40: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'llvm::ArrayRef<clang::IdentifierLoc> &&' for 1st argument
/Users/ec2-user/jenkins/workspace/llvm.org/as-lldb-cmake/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:70:18: note: candidate constructor not viable: no known conversion from 'std::pair<clang::IdentifierInfo *, clang::SourceLocation>' to 'std::nullopt_t' for 1st argument
/*implicit*/ ArrayRef(std::nullopt_t) {}
```
I found this issue when I working on
https://github.com/llvm/llvm-project/pull/107168.
Currently we have many similiar data structures like:
- `std::pair<IdentifierInfo *, SourceLocation>`.
- Element type of `ModuleIdPath`.
- `IdentifierLocPair`.
- `IdentifierLoc`.
This PR unify these data structures to `IdentifierLoc`, moved
`IdentifierLoc` definition to SourceLocation.h, and deleted other
similer data structures.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
When we mark a module visible, we normally mark all of its non-explicit
submodules and other exports as visible. However, when we first enter a
submodule we should not make them visible to the submodule itself until
they are actually imported. Marking exports visible before import would
cause bizarre behaviour with local submodule visibility, because it
happened before we discovered the submodule's transitive imports and
could fail to make them visible in the parent module depending on
whether the submodules involved were explicitly defined (module X) or
implicitly defined from an umbrella (module *).
rdar://136524433
Consider the following:
```
template<typename T>
struct A { };
template<typename T>
int A<T>::B::* f(); // error: no member named 'B' in 'A<T>'
```
Although this is clearly valid, clang rejects it because the
_nested-name-specifier_ `A<T>::` is parsed as-if it was declarative,
meaning, we parse it as-if it was the _nested-name-specifier_ in a
redeclaration/specialization. However, we don't (and can't) know whether
the _nested-name-specifier_ is declarative until we see the '`*`' token,
but at that point we have already complained that `A` has no member
named `B`! This patch addresses this bug by adding support for _fully_
unannotated _and_ unbounded tentative parsing, which allows for us to
parse past tokens without having to cache them until we reach a point
where we can guarantee to be past the construct we are disambiguating.
I don't know where the approach taken here is ideal -- alternatives are
welcome. However, the performance impact (as measured by
llvm-compile-time-tracker (https://llvm-compile-time-tracker.com/?config=Overview&stat=instructions%3Au&remote=sdkrystian)
is quite minimal (0.09%, which I plan to further improve).
This PR implement [P3034R1 Module Declarations Shouldn’t be
Macros](https://wg21.link/P3034R1), and refactor the convoluted state
machines in module name lexical analysis.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Co-authored-by: Aaron Ballman <aaron@aaronballman.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
The commit adds serialization and de-serialization implementations for
the stored regions. Basically, the serialized representation of the
regions of a PP is a (ordered) sequence of source location encodings.
For de-serialization, regions from loaded files are stored by their ASTs.
When later one queries if a loaded location L is in an opt-out
region, PP looks up the regions of the loaded AST where L is at.
(Background if helps: a pair of `#pragma clang unsafe_buffer_usage begin/end` pragmas marks a
warning-opt-out region. The begin and end locations (opt-out regions)
are stored in preprocessor instances (PP) and will be queried by the
`-Wunsafe-buffer-usage` analyzer.)
The reported issue at upstream: https://github.com/llvm/llvm-project/issues/90501
rdar://124035402
Add some primitive syntax highlighting to our code snippet output.
This adds "checkpoints" to the Preprocessor, which we can use to start lexing from. When printing a code snippet, we lex from the nearest checkpoint and highlight the tokens based on their token type.
Previous representation used an enumeration combined to a switch to
dispatch to the appropriate lexer.
Use function pointer so that the dispatching is just an indirect call,
which is actually better because lexing is a costly task compared to a
function call.
This also makes the code slightly cleaner, speedup on compile time
tracker are consistent and range form -0.05% to -0.20% for NewPM-O0-g,
see
https://llvm-compile-time-tracker.com/compare.php?from=f9906508bc4f05d3950e2219b4c56f6c078a61ef&to=608c85ec1283638db949d73e062bcc3355001ce4&stat=instructions:u
Considering just the preprocessing task, preprocessing the sqlite
amalgametion takes -0.6% instructions (according to valgrind
--tool=callgrind)
---------
Co-authored-by: serge-sans-paille <sguelton@mozilla.com>
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
`ModuleDeclState` is incorrectly changed to `NamedModuleImplementation`
for `struct module {}; void foo(module a);`. This is mostly benign but
leads to a spurious warning after #69555.
A real world example is:
```
// pybind11.h
class module_ { ... };
using module = module_;
// tensorflow
void DefineMetricsModule(pybind11::module main_module);
// `module main_module);` incorrectly changes `ModuleDeclState` to `NamedModuleImplementation`
#include <algorithm> // spurious warning
```
This fixes many unit tests when trying to enable IncrementalExtensions
by default for testing purposes.
Differential Revision: https://reviews.llvm.org/D158415
This new method repeatedly calls Lex() until end of file is reached
and optionally fills a std::vector of Tokens. Use it in Clang's unit
tests to avoid quite some code duplication.
Differential Revision: https://reviews.llvm.org/D158413
The preprocessor `IncrementalProcessing` option was being used to
control whether or not to teardown the lexer or run the end of
translation unit action. In D127284 this was merged with
`-fincremental-extensions`, which also changes top level parsing.
Split these again so that the former behavior can be achieved without
the latter (ie. to allow managing lifetime without also changing
parsing).
Resolves rdar://113406310.
Setting __FLT_EVAL_METHOD__ to -1 with fast-math will set
__GLIBC_FLT_EVAL_METHOD to 2 and long double ends up being used for
float_t and double_t. This creates some ABI breakage with various C libraries.
See details here: https://github.com/llvm/llvm-project/issues/60781
This reverts commit bbf0d1932a3c1be970ed8a580e51edf571b80fd5.
As the diagnostic message shows, we should remove -fmodules-ts flag in
clang/llvm17. Since clang/llvm16 is already branched. We can remove the
depreacared flag now.
This patch prepares the necessary interfaces in the preprocessor part
for D137527 since we need to recognize if we're in a module unit, the
module kinds and the module declaration and the module we're importing
in the preprocessor.
Differential Revision: https://reviews.llvm.org/D137526
Add a pair of clang pragmas:
- `#pragma clang unsafe_buffer_usage begin` and
- `#pragma clang unsafe_buffer_usage end`,
which specify the start and end of an (unsafe buffer checking) opt-out
region, respectively.
Behaviors of opt-out regions conform to the following rules:
- No nested nor overlapped opt-out regions are allowed. One cannot
start an opt-out region with `... unsafe_buffer_usage begin` but never
close it with `... unsafe_buffer_usage end`. Mis-use of the pragmas
will be warned.
- Warnings raised from unsafe buffer operations inside such an opt-out
region will always be suppressed. This behavior CANNOT be changed by
`clang diagnostic` pragmas or command-line flags.
- Warnings raised from unsafe operations outside of such opt-out
regions may be reported on declarations inside opt-out
regions. These warnings are NOT suppressed.
- An un-suppressed unsafe operation warning may be attached with
notes. These notes are NOT suppressed as well regardless of whether
they are in opt-out regions.
The implementation maintains a separate sequence of location pairs
representing opt-out regions in `Preprocessor`. The `UnsafeBufferUsage`
analyzer reads the region sequence to check if an unsafe operation is
in an opt-out region. If it is, discard the warning raised from the
operation immediately.
This is a re-land after I reverting it at 9aa00c8a306561c4e3ddb09058e66bae322a0769.
The compilation error should be resolved.
Reviewed by: NoQ
Differential revision: https://reviews.llvm.org/D140179
Add a pair of clang pragmas:
- `#pragma clang unsafe_buffer_usage begin` and
- `#pragma clang unsafe_buffer_usage end`,
which specify the start and end of an (unsafe buffer checking) opt-out
region, respectively.
Behaviors of opt-out regions conform to the following rules:
- No nested nor overlapped opt-out regions are allowed. One cannot
start an opt-out region with `... unsafe_buffer_usage begin` but never
close it with `... unsafe_buffer_usage end`. Mis-use of the pragmas
will be warned.
- Warnings raised from unsafe buffer operations inside such an opt-out
region will always be suppressed. This behavior CANNOT be changed by
`clang diagnostic` pragmas or command-line flags.
- Warnings raised from unsafe operations outside of such opt-out
regions may be reported on declarations inside opt-out
regions. These warnings are NOT suppressed.
- An un-suppressed unsafe operation warning may be attached with
notes. These notes are NOT suppressed as well regardless of whether
they are in opt-out regions.
The implementation maintains a separate sequence of location pairs
representing opt-out regions in `Preprocessor`. The `UnsafeBufferUsage`
analyzer reads the region sequence to check if an unsafe operation is
in an opt-out region. If it is, discard the warning raised from the
operation immediately.
Reviewed by: NoQ
Differential revision: https://reviews.llvm.org/D140179
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
