Consider the following code:
```cpp
# 1 __FILE__ 1 3
export module a;
```
According to the wording in
[P1857R3](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1857r3.html):
```
A module directive may only appear as the first preprocessing tokens in a file (excluding the global module fragment.)
```
and the wording in
[[cpp.pre]](https://eel.is/c++draft/cpp.pre#nt:module-file)
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
`#` is the first pp-token in the translation unit, and it was rejected
by clang, but they really should be exempted from this rule. The goal is
to not allow any preprocessor conditionals or most state changes, but
these don't fit that.
State change would mean most semantically observable preprocessor state,
particularly anything that is order dependent. Global flags like being a
system header/module shouldn't matter.
We should exempt a brunch of directives, even though it violates the
current standard wording.
In this patch, we introduce a `TrivialDirectiveTracer` to trace the
**State change** that described above and propose to exempt the
following kind of directive: `#line`, GNU line marker, `#ident`,
`#pragma comment`, `#pragma mark`, `#pragma detect_mismatch`, `#pragma
clang __debug`, `#pragma message`, `#pragma GCC warning`, `#pragma GCC
error`, `#pragma gcc diagnostic`, `#pragma OPENCL EXTENSION`, `#pragma
warning`, `#pragma execution_character_set`, `#pragma clang
assume_nonnull` and builtin macro expansion.
Fixes https://github.com/llvm/llvm-project/issues/145274
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
Handles clang::DiagnosticsEngine and clang::DiagnosticIDs.
For DiagnosticIDs, this mostly migrates from `new DiagnosticIDs` to
convenience method `DiagnosticIDs::create()`.
Part of cleanup https://github.com/llvm/llvm-project/issues/151026
This switches to `makeIntrusiveRefCnt<FileSystem>` where creating a new
object, and to passing/returning by `IntrusiveRefCntPtr<FileSystem>`
instead of `FileSystem*` or `FileSystem&`, when dealing with existing
objects.
Part of cleanup #151026.
Previously, the newline after a module directive was not properly
captured and printed by `clang::printDependencyDirectivesAsSource`.
According to P1857R3, each directive must, after skipping horizontal
whitespace, appear at the start of a logical line. Because the newline
after module directives was missing, this invalidated the following
line.
This fixes tests that were previously in violation of P1857R3,
including for Objective-C directives, which should also comply with
P1857R3.
This also ensures that the global module fragment `module;` is captured
by the dependency directives scanner.
The dependency directive scanner was incorrectly classifying namespaces
such as `import::inner xi` as directives. According to P1857R3, `import` should
not be treated as a directive when followed by `::`.
This change fixes that behavior.
The `SourceLocation` of a `RootSignatureToken` is incorrectly set to be
the "offset" into the concatenated string that denotes the
rootsignature. This causes an issue when the `StringLiteral` is a
multi-line expansion macro, since the offset will not account for the
characters between `StringLiteral` tokens.
This pr resolves this by retaining the `SourceLocation` information that
is kept in `StringLiteral` and then converting the offset in the
concatenated string into the proper `SourceLocation` using the
`StringLiteral::getLocationOfByte` interface. To do so, we will need to
adjust the `RootSignatureToken` to only hold its offset into the root
signature string. Then when the parser will use the token, it will need
to compute its actual `SourceLocation`.
See linked issue for more context.
For example:
```
#define DemoRootSignature \
"CBV(b0)," \
"RootConstants(num32BitConstants = 3, b0, invalid)"
expected caret location ---------------^
actual caret location ------------^
```
The caret points 5 characters early because the current offset did not
account for the characters:
```
'"' ' ' '\' ' ' '"'
1 2 3 4 5
```
- Updates `RootSignatureParser` to retain `SourceLocation` information
by retaining the `StringLiteral` and passing the underlying `StringRef`
to the `Lexer`
- Updates `RootSignatureLexer` so that the constructed tokens only
reflect an offset into the `StringRef`
- Updates `RootSignatureParser` to directly construct its used `Lexer`
so that the `StringLiteral` is directly tied with the string used in the
`RootSignatureLexer`
- Updates `RootSignatureParser` to use
`StringLiteral::getLocationOfByte` to get the actual token location for
diagnostics
- Updates `ParseHLSLRootSignatureTest` to construct a phony
`AST`/`StringLiteral` for the test cases
- Adds a test to `RootSignature-err.hlsl` showing that the
`SourceLocation` is correctly set for diagnostics in a multi-line macro
expansion
Resolves: https://github.com/llvm/llvm-project/issues/146967
`clang-scan-deps` threw "unterminated conditional directive" error
falsely on the following example:
```
#ifndef __TEST
#define __TEST
#if defined(__TEST_DUMMY)
#if defined(__TEST_DUMMY2)
#pragma GCC warning \
"Hello!"
#else
#pragma GCC error \
"World!"
#endif // defined(__TEST_DUMMY2)
#endif // defined(__TEST_DUMMY)
#endif // #ifndef __TEST
```
The issue comes from PR #143950, where the flag `LastNonWhitespace` does
not correctly represent the state for the example above. The PR aimed to
support that a line-continuation can be followed by whitespaces.
This commit fixes the issue by moving the `LastNonWhitespace` variable
to the inner loop so that it will be correctly reset.
rdar://153742186
Introduce a type alias for the commonly used `std::pair<FileID,
unsigned>` to improve code readability, and make it easier for future
updates (64-bit source locations).
Depends on [[clang][Preprocessor] Add peekNextPPToken, makes look ahead
next token without
side-effects](https://github.com/llvm/llvm-project/pull/143898).
This PR fix the performance regression that introduced in
https://github.com/llvm/llvm-project/pull/144233.
The original PR(https://github.com/llvm/llvm-project/pull/144233) handle
the first pp-token in the main source file in the macro
definition/expansion and `Lexer::Lex`, but the lexer is almost always on
the hot path, we may hit a performance regression. In this PR, we handle
the first pp-token in `Preprocessor::EnterMainSourceFile`.
---------
Signed-off-by: yronglin <yronglin777@gmail.com>
This PR is 2nd part of
[P1857R3](https://github.com/llvm/llvm-project/pull/107168)
implementation, and mainly implement the restriction `A module directive
may only appear as the first preprocessing tokens in a file (excluding
the global module fragment.)`:
[cpp.pre](https://eel.is/c++draft/cpp.pre):
```
module-file:
pp-global-module-fragment[opt] pp-module group[opt] pp-private-module-fragment[opt]
```
We also refine tests use `split-file` instead of conditional macro.
Signed-off-by: yronglin <yronglin777@gmail.com>
P2223R2 allows the line-continuation slash `\` to be followed by
additional whitespace. The Clang lexer already follows this behavior,
also for versions prior to C++23. The dependency directive scanner
however only implements it for `#define` directives (15d5f5d).
This fully implements P2223R2 for the dependency directive scanner (for
any C++ standard) and aligns the dependency directive scanner's splicing
behavior with that of the Clang lexer.
For example, the following code was previously not scanned correctly by
`clang-scan-deps` but now works as expected:
```cpp
import \<whitespace here>
A;
```
Sometimes, when a user writes invalid code, the minimization used for
scanning can create a stream of tokens that is invalid at lex time. This
patch protects against the case where there are valid (non-c++20) import
directives discovered in the middle of an invalid `import` declaration.
Mostly authored by: @akyrtzi
resolves: rdar://152335844
- defines in-memory reprsentation of `comparisonFunc` and `borderColor`
- defines parsing of the `ComparisonFunc` and `StaticBorderColor` enum
- integrates parsing of these number parameters with their respective
`parseComparisonFunc` and `parseStaticBorderColor`
- adds basic unit tests to demonstrate setting functionality
Part 6 of https://github.com/llvm/llvm-project/issues/126574
The current naming of the `enum class Filter` and the Filter struct
member causes ambiguity.
This change will be reverted to be addressed by renaming the variable.
Reverts llvm/llvm-project#140294
- defines in-memory reprsentation of `filter`
- defines parsing of the `Filter` enum
- integrates parsing of these number parameters with their respective,
`parseFilter`
- adds basic unit tests to demonstrate setting functionality
Part 5 of https://github.com/llvm/llvm-project/issues/126574
- defines in-memory reprsentation of `address[U|V|W]`
- defines parsing of the `TextureAddressMode` enum
- integrates parsing of these number parameters with their respective,
`parseTextureAddressMode`
- adds basic unit tests to demonstrate setting functionality
Part 4 of https://github.com/llvm/llvm-project/issues/126574
- defines in-memory reprsentation of `maxAnisotropy`, `minLOD` and
`maxLOD`
- integrates parsing of these number parameters with their respective,
`parseUInt` and `parseFloat` respectively
- adds basic unit tests to demonstrate setting functionality
Part 3 of https://github.com/llvm/llvm-project/issues/126574
- defines in-memory representaiton of MipLODBias to allow for testing of
a float parameter
- defines `handleInt` and `handleFloat` to handle converting a token's
`NumSpelling` into a valid float
- plugs this into `parseFloatParam` to fill in the MipLODBias param
The parsing of floats is required to match the behaviour of DXC. This
behaviour is outlined as follows:
- if the number is an integer then convert it using `_atoi64`, check for
overflow and static_cast this to a float
- if the number is a float then convert it using `strtod`, check for
float overflow and static_cast this to a float, this will implicitly
also check for double over/underflow and if the string is malformed then
it will return an error
This pr matches this behaviour by parsing as, uint/int accordingly and
then casting, or, by using the correct APFloat semantics/rounding mode
with `NumericLiteralParser`.
- adds testing of error diagnostics and valid float param values to
demonstrate functionality
Part 2 of https://github.com/llvm/llvm-project/issues/126574
- define StaticSampler in-memory representation
- implement the infastructure for parsing parameters of StaticSampler
- define and implement parsing of the `s` reg to demonstrate
functionality
- add unit tests
First part of https://github.com/llvm/llvm-project/issues/126574
This reverts commit e2a885537f11f8d9ced1c80c2c90069ab5adeb1d. Build failures were fixed right away and reverting the original commit without the fixes breaks the build again.
The `DiagnosticOptions` class is currently intrusively
reference-counted, which makes reasoning about its lifetime very
difficult in some cases. For example, `CompilerInvocation` owns the
`DiagnosticOptions` instance (wrapped in `llvm::IntrusiveRefCntPtr`) and
only exposes an accessor returning `DiagnosticOptions &`. One would
think this gives `CompilerInvocation` exclusive ownership of the object,
but that's not the case:
```c++
void shareOwnership(CompilerInvocation &CI) {
llvm::IntrusiveRefCntPtr<DiagnosticOptions> CoOwner = &CI.getDiagnosticOptions();
// ...
}
```
This is a perfectly valid pattern that is being actually used in the
codebase.
I would like to ensure the ownership of `DiagnosticOptions` by
`CompilerInvocation` is guaranteed to be exclusive. This can be
leveraged for a copy-on-write optimization later on. This PR changes
usages of `DiagnosticOptions` across `clang`, `clang-tools-extra` and
`lldb` to not be intrusively reference-counted.
- defines the `RootFlags` in-memory enum
- defines `parseRootFlags` to parse the various flag enums into a single
`uint32_t`
- adds corresponding unit tests
- improves the diagnostic message for when we provide a non-zero integer
value to the flags
Resolves https://github.com/llvm/llvm-project/issues/126575
- defines the empty RootConstants in-memory struct
- adds test harness for testing it
- adds missing parameter keywords to the lexer (`RootConstants`,
`num32BitConstants`)
First part of implementing:
https://github.com/llvm/llvm-project/issues/126576
Restore the behavior existing prior to
fe2eefc4718f57e1753f7bd51c158fc03d70b34f. Make reporting of unevaluated
directive source range more consistent and with fewer assumptions. In
case of a failed evaluation don't assume any specific token and don't
assume correct `PPValue` range tracking.
- this takes care to add support to match the [behaviour of
DXC](34b6d0f91e/tools/clang/lib/Parse/HLSLRootSignature.cpp (L74))
acceptable floating point integers
Namely:
- Allow for specifying the decimal '.'
- Allow for specifying exponents with 'e' or 'E' and allow for 'f' to
denote an otherwise interpreted integer as a float
This pr is simply responsible of creating a token that could be
interpeted as a floating point integer by `NumericLiteralParser`. As
such, we are not required to validate that the special characters only
occur once and that 'f' is only at the end of the string. These will be
validated when invoking `NumericLiteralParser` during parsing.
Resolves#126565
This PR hides the reference-counted pointer that holds `TargetOptions`
from the public API of `CompilerInvocation`. This gives
`CompilerInvocation` an exclusive control over the lifetime of this
member, which will eventually be leveraged to implement a copy-on-write
behavior.
There are two clients that currently share ownership of that pointer:
* `TargetInfo` - This was refactored to hold a non-owning reference to
`TargetOptions`. The options object is typically owned by the
`CompilerInvocation` or by the new `CompilerInstance::AuxTargetOpts` for
the auxiliary target. This needed a bit of care in `ASTUnit::Parse()` to
keep the `CompilerInvocation` alive.
* `clangd::PreambleData` - This was refactored to exclusively own the
`TargetOptions` that get moved out of the `CompilerInvocation`.
- defines the special values for `DESCRIPTOR_RANGE_OFFSET_APPEND` and
`unbounded` for the `offset` and `numDescriptors` parameters
respectively
- adds these parmaters to the `DescriptorClause` struct and the params
struct
- plugs in parsing of `numDescriptors` and `offset` into
`parseDescriptorTableClauseParams`
- defines the `unbounded` enum keyword for the lexer to expose to the
parser
- adds corresponding unit tests
Part 5 of #126569
This PR fixes two issues in one go:
1. The dependency directives getter (a `std::function`) was being stored
in `PreprocessorOptions`. This goes against the principle where the
options classes are supposed to be value-objects representing the `-cc1`
command line arguments. This is fixed by moving the getter directly to
`CompilerInstance` and propagating it explicitly.
2. The getter was capturing the `ScanInstance` VFS. That's fine in
synchronous implicit module builds where the same VFS instance is used
throughout, but breaks down once you try to build modules asynchronously
(which forces the use of separate VFS instances). This is fixed by
explicitly passing a `FileManager` into the getter and extracting the
right instance of the scanning VFS out of it.
- when developing the RootSignatureLexer library, we are creating new
files so we should set the standard to adhere to the coding conventions
for function naming
- this was missed in the initial review but caught in the review of the
parser pr
[here](https://github.com/llvm/llvm-project/pull/133302#discussion_r2017632092)
Co-authored-by: Finn Plummer <finnplummer@microsoft.com>
This reapplies 5ffd9bdb50b57 (#133545) with fixes.
The BUILD_SHARED_LIBS=ON build was fixed by adding missing LLVM
dependencies to the InterpTests binary in
unittests/AST/ByteCode/CMakeLists.txt .
This pr relands https://github.com/llvm/llvm-project/pull/133302.
It resolves two issues:
- Linking error during build,
[here](https://github.com/llvm/llvm-project/pull/133302#issuecomment-2767259848).
There was a missing dependency for `clangLex` for the
`ParseHLSLRootSignatureTest.cpp` unit testing. This library was added to
the dependencies to resolve the error. It wasn't caught previously as
the library was transitively linked in most build environments
- Warning of unused declaration,
[here](https://github.com/llvm/llvm-project/pull/133302#issuecomment-2767091368).
There was a usability line in `LexHLSLRootSignature.h` of the form
`using TokenKind = enum RootSignatureToken::Kind` which causes this
error. The declaration is removed from the header file to be used
locally in the `.cpp` files that use it.
Notably, the original pr would also exposed `clang::hlsl::TokenKind` to
everywhere it was included, which had a name clash with
`tok::TokenKind`. This is another motivation to change to the proposed
resolution.
---------
Co-authored-by: Finn Plummer <finnplummer@microsoft.com>
Pass all the dependencies into add_clang_unittest. This is consistent
with how it is done for LLDB. I borrowed the same named argument list
structure from add_lldb_unittest. This is a necessary step towards
consolidating unit tests into fewer binaries, but seems like a good
refactoring in its own right.
Reverts llvm/llvm-project#133302
Reverting to inspect build failures that were introduced from use of the
`clang::Preprocessor` in unit testing, as well as, the warning about an
unused declaration. See linked issue for failures.
- defines the Parser class and an initial set of helper methods to
support consuming tokens. functionality is demonstrated through a simple
empty descriptor table test case
- defines an initial in-memory representation of a DescriptorTable
- implements a test harness that will be used to validate the correct
diagnostics are generated. it will construct a dummy pre-processor with
diagnostics consumer to do so
Implements the first part of
https://github.com/llvm/llvm-project/issues/126569
From the corrections to the Root Signature specification here:
https://github.com/llvm/wg-hlsl/issues/192. It was denoted that keywords
are also case-insensitive in DXC.
This pr adjusts the lexer to adhere to the updated spec.
We also have a NFC to add a missing license to a file while in the area.
---------
Co-authored-by: Finn Plummer <finnplummer@microsoft.com>
This makes it so that `CompilerInvocation` can be the only entity that
manages ownership of `HeaderSearchOptions`, making it possible to
implement copy-on-write semantics.
This PR adds new `ModuleCache` interface to Clang's implicitly-built
modules machinery. The main motivation for this change is to create a
second implementation that uses a more efficient kind of
`llvm::AdvisoryLock` during dependency scanning.
In addition to the lock abstraction, the `ModuleCache` interface also
manages the existing `InMemoryModuleCache` instance. I found that
compared to keeping these separate/independent, the code is a bit
simpler now, since these are two tightly coupled concepts. I can
envision a more efficient implementation of the `InMemoryModuleCache`
for the single-process case too, which will be much easier to implement
with the current setup.
This is not intended to be a functional change.
For the sake of scope, we will let the lexing of floating literals be
deferred until needed for Static Samplers. Other than that this pr
should allow us to simply define new enumerations/keywords in
`RootSignatureTokenKinds.def` for when they are used in the parser. We
could have defined all of these keywords here, but for the sake of
correctness in review we will let them be split up.
- Define `RootSignatureLexer` and provide a public `LexToken` method for
external use
- Define the file `RootSignatureTokenKinds` to define required tokens
and allow for future custom keywords/enums
- Implement the internal methods required to parse the different types
of tokens (integers, flag enums, puncuators...)
- Add test harness for unit testing and the respective unit tests for
lexing the tokens
Resolves#126563
---------
Co-authored-by: Chris B <beanz@abolishcrlf.org>
Similarly to https://github.com/llvm/llvm-project/pull/122918, leading
comments are currently not being moved.
```
struct Foo {
// This one is the cool field.
int a;
int b;
};
```
becomes:
```
struct Foo {
// This one is the cool field.
int b;
int a;
};
```
but should be:
```
struct Foo {
int b;
// This one is the cool field.
int a;
};
```
We have two copies of the same code in clang-tidy and
clang-reorder-fields, and those are extremenly similar to
`Lexer::findNextToken`, so just add an extra agument to the latter.
---------
Co-authored-by: cor3ntin <corentinjabot@gmail.com>
The Lexer used in getRawToken is not told to keep whitespace, so when it
skips over escaped newlines, it also ignores whitespace, regardless of
getRawToken's IgnoreWhiteSpace parameter.
Instead of letting this case fall through to lexing, check
for whitespace after skipping over any escaped newlines.
This PR removes the `HeaderFileInfo::Framework` member and reduces the
size of this data type from 32B to 16B. This should improve Clang's
memory usage in situations where it keeps track of lots of header files.
NFCI. Depends on #114459.
This PR removes the `-index-header-map` functionality from Clang. AFAIK
this was only used internally at Apple and is now dead code. The main
motivation behind this change is to enable the removal of
`HeaderFileInfo::Framework` member and reducing the size of that data
structure.
rdar://84036149
Follow-up to `34ab855826b8cb0c3b46c770b83390bd1fe95c64`:
* Don't fail the scan with an include with missing filename, it may be
inside a skipped preprocessor block. Let the compilation provide any
related error.
* Fix an issue where the lexer was skipping through the next directive,
after ignoring the include with missing filename.
Previously source input like `#import ` resulted in infinite calls
append the same token into `CurDirTokens`. This patch now ignores those
directive lines if they won't actually end up being compiled. (e.g.
macro guarded)
resolves: rdar://121247565
`import:(type)name` is a method argument decl in ObjC, but the C++20
preprocessing rules say this is a preprocessing line.
Because the dependency directive scanner is not language dependent, this
patch extends the C++20 rule to exclude `module :` (which is never a
valid module decl anyway), and `import :` that is not followed by an
identifier.
This is ok to do because in C++20 mode the compiler will later error on
lines like this anyway, and the dependencies the scanner returns are
still correct.
