44 Commits

Author SHA1 Message Date
Eduardo Caldas
7b404b6d00 Add FloatingLiteral to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82318
2020-06-25 17:05:08 +00:00
Eduardo Caldas
466e8b7ea6 Add StringLiteral to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82360
2020-06-25 17:05:08 +00:00
Eduardo Caldas
221d7bbe49 Add CharLiteral to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82312
2020-06-25 17:05:08 +00:00
Eduardo Caldas
7f7f856480 Add BoolLiteralExpression to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82310
2020-06-25 15:37:53 +00:00
Eduardo Caldas
1b2f6b4a08 Add support for DeclRefExpr in SyntaxTree, by generating IdExpressions
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: hlopko, gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81168
2020-06-18 21:05:23 +02:00
Shengchen Kan
2c63ea6ede [TEST] TreeTest.cpp - Add a comma to avoid build error with -werror
Summary:
The macro `INSTANTIATE_TEST_CASE_P` is defined as
```
\# define INSTANTIATE_TEST_CASE_P(prefix, test_case_name, generator, ...) \
...
```

If we build the test case with -werror, we will get an error like
```
error: ISO C++11 requires at least one argument for the "..." in a
variadic macro

testing::ValuesIn(TestClangConfig::allConfigs()));
                                                ^
```
This patch fixes that.

Reviewers: gribozavr, hlopko, eduucaldas, gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81388
2020-06-08 23:14:16 +08:00
Eduardo Caldas
3b739690b0 Add support for IntegerLiteral in SyntaxTree
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81135
2020-06-04 14:05:31 +02:00
Dmitri Gribenko
06cf7adcc8 Make syntax tree test print the line number when it fails
Summary:
The syntax tree test uses a helper function that executes all testing
assertions. When an assertion fails, the only line number that gets
printed to the log refers to the helper function. After this change, we
would also get the line number of the EXPECT_TRUE macro invocation
(unfortunately, the line number of the last token of it, not the first
one, but there's not much I can do about it).

Reviewers: hlopko, eduucaldas

Reviewed By: hlopko, eduucaldas

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81107
2020-06-04 10:42:44 +02:00
Eduardo Caldas
007098d7e6 Add support for nullptr in SyntaxTrees
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81092
2020-06-03 15:52:33 +02:00
Dmitri Gribenko
d7d5dd31fc Split syntax tree tests into more granular ones
Summary:
Doing so allows us to increase test coverage by removing unnecessary
language restrictions.

Reviewers: hlopko, eduucaldas

Reviewed By: hlopko, eduucaldas

Subscribers: gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81040
2020-06-03 11:17:11 +02:00
Dmitri Gribenko
b34b7691fa Syntax tree: ignore implicit expressions at the top level of statements
Summary:
I changed `markStmtChild` to ignore implicit expressions the same way as
`markExprChild` does it already. The test that I modified crashes
without this change.

Reviewers: hlopko, eduucaldas

Reviewed By: hlopko, eduucaldas

Subscribers: gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81019
2020-06-03 10:58:12 +02:00
Dmitri Gribenko
53c29a42d0 Reinstate the syntax tree test for 'static' in an array subscript
Reviewers: eduucaldas

Reviewed By: eduucaldas

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81009
2020-06-02 17:01:51 +02:00
Dmitri Gribenko
d559185aae Renamed Lang_C to Lang_C99, Lang_CXX to Lang_CXX03, and 2a to 20
Summary:
I think we would be better off with tests explicitly specifying the
language mode. Right now Lang_C means C99, but reads as "any C version",
or as "unspecified C version".

I also changed '-std=c++98' to '-std=c++03' because they are aliases (so
there is no difference in practice), because Clang implements C++03
rules in practice, and because 03 makes a nice sortable progression
between 03, 11, 14, 17, 20.

Reviewers: shafik, hlopko

Reviewed By: hlopko

Subscribers: jfb, martong, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81000
2020-06-02 16:31:20 +02:00
Dmitri Gribenko
44f989e780 Run syntax tree tests in many language modes
Reviewers: hlopko, eduucaldas

Reviewed By: hlopko, eduucaldas

Subscribers: gribozavr2, mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80822
2020-06-02 10:30:01 +02:00
Eduardo Caldas
3a574a6cb3 Add support for Overloaded Binary Operators in SyntaxTree
Reviewers: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80812
2020-05-29 20:03:59 +02:00
Eduardo Caldas
eca41919d2 Improve test infrastructure in SyntaxTree
Summary:
* Test if the code sourcing the SyntaxTree compiles
* Output compiler errors and warnings to err
* Fix tests with code that did not compile

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80731
2020-05-28 21:35:12 +02:00
Eduardo Caldas
461af57de7 Add support for UnaryOperator in SyntaxTree
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80624
2020-05-27 17:12:46 +02:00
Eduardo Caldas
3785eb83af Add support for binary operators in Syntax Trees
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80540
2020-05-26 12:25:58 +02:00
Marcel Hlopko
1bf055c989 [Syntax] Add mapping from spelled to expanded tokens for TokenBuffer
Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can be mapped.

Taking over from https://reviews.llvm.org/D72581.

Reviewers: gribozavr2, sammccall

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77209
2020-04-07 15:07:16 +02:00
Marcel Hlopko
cdce2fe561 [Syntax] Remove delayed folding from tree building.
Summary:
This patch removes delayed folding and replaces it with forward peeking.

Delayed folding was previously used as a solution to the problem that
declaration doesn't have a representation in the AST. For example following
code:

```
int a,b;
```

is expressed in the AST as:

```
TranslationUnitDecl
|-...
|-VarDecl `int a`
`-VarDecl `int b`
```

And in the syntax tree we need:

```
*: TranslationUnit
`-SimpleDeclaration
  |-int
  |-SimpleDeclarator
  | `-a
  |-,
  |-SimpleDeclarator
  | `-b
  |-;
```

So in words, we need to create SimpleDeclaration to be a parent of
SimpleDeclarator nodes. Previously we used delayed folding to make sure SimpleDeclarations will be
eventually created. And in case multiple declarators requested declaration
creation, declaration range was extended to cover all declarators.

This design started to be hard to reason about, so we decided to replace it with
forward peeking. The last declarator node in the chain is responsible for creating
SimpleDeclaration for the whole chain. Range of the declaration corresponds to
the source range of the declarator node. Declarator decides whether its the last
one by peeking to the next AST node (see `isResponsibleForCreatingDeclaration`).

This patch does following:
* Removed delayed folding logic
* Tweaks Token.dumpForTests
* Moves getQualifiedNameStart inside BuildTreeVisitor
* Extracts BuildTreeVisitor.ProcessDeclaratorAndDeclaration
* Renames Builder.getDeclRange to Builder.getDeclarationRange and uses the
  method in all places.
* Adds a bunch of tests

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76922
2020-03-31 09:47:50 +02:00
Marcel Hlopko
eddede9d51 [Syntax] Test both the default and windows target platforms in unittests
Summary:
This increases the coverage for things that differ between Linux and Windows, such as -fdelayed-template-parsing. This would have prevented the rollback of https://reviews.llvm.org/D76346.

While at it, update -std=c++11 to c++17 for the test.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76497
2020-03-20 16:33:58 +01:00
Dmitri Gribenko
9967352a03 Revert "[Syntax] Test both the default and windows target platforms in unittests"
This reverts commit fd7300f717c18c861e77685efe6f16f12fb63ae7. The fix
in this patch didn't help and the Windows buildbot broke:
http://45.33.8.238/win/10881/step_7.txt
2020-03-20 12:13:49 +01:00
Marcel Hlopko
fd7300f717 [Syntax] Test both the default and windows target platforms in unittests
Summary:
This increases the coverage for things that differ between Linux and Windows, such as `-fdelayed-template-parsing`. This would have prevented the rollback of https://reviews.llvm.org/D76346.

While at it, update -std=c++11 to c++17 for the test.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76433
2020-03-20 11:42:18 +01:00
Marcel Hlopko
e9630630ff [Syntax] Split syntax tests
Summary:
This patch split Basic test into multple individual tests to allow simpler
filtering and clearer signal into what's broken when it's broken.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76366
2020-03-20 09:53:43 +01:00
Marcel Hlopko
88bf9b3d26 [Syntax] Build template declaration nodes
Summary:
Rollforward of
https://reviews.llvm.org/rGdd12826808f9079e164b82e64b0697a077379241 after
temporarily adding -fno-delayed-template-parsing to the TreeTest.

Original summary:

> Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
>
> Handles template declaration of all kinds.
>
> Also builds template declaration nodes for specializations and explicit
> instantiations of classes.
>
> Some missing things will be addressed in the follow-up patches:
>
> * specializations of functions and variables,
> * template parameters.

Reviewers: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76418
2020-03-19 17:43:07 +01:00
Nico Weber
881f5b5a7b Revert "[Syntax] Build template declaration nodes"
This reverts commit dd12826808f9079e164b82e64b0697a077379241.
Breaks tests on Windows, see https://reviews.llvm.org/D76346#1929208
2020-03-18 12:57:55 -04:00
Marcel Hlopko
dd12826808 [Syntax] Build template declaration nodes
Summary:
Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.

Handles template declaration of all kinds.

Also builds template declaration nodes for specializations and explicit
instantiations of classes.

Some missing things will be addressed in the follow-up patches:

specializations of functions and variables,
template parameters.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76346
2020-03-18 16:16:59 +01:00
Marcel Hlopko
7d382dcd46 [Syntax] Build declarator nodes
Summary:
Copy of https://reviews.llvm.org/D72089 with Ilya's permission. See
https://reviews.llvm.org/D72089 for the first batch of comments.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76220
2020-03-16 19:13:59 +01:00
Benjamin Kramer
adcd026838 Make llvm::StringRef to std::string conversions explicit.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.
2020-01-28 23:25:25 +01:00
Ilya Biryukov
013c07f697 [Syntax] Unset IsOriginal flag on nodes removed from the tree
And add a corresponding test.
Only nodes inside the TranslationUnit subtree can be marked as original,
computeReplacements() relies on this.
2020-01-14 17:00:33 +01:00
Ilya Biryukov
07a41018e9 [Syntax] Mark synthesized nodes as modifiable
This was an oversight in the original patch.
Also add corresponding tests.
2020-01-14 16:41:09 +01:00
Ilya Biryukov
04f627f6b9 [Syntax] Build spanning SimpleDecalration for classes, structs, etc
When they are free-standing, e.g. `struct X;` or `struct X {};`.
Although this complicates the common case (of free-standing class
declarations), this ensures the less common case (e.g. `struct X {} a;`)
are handled uniformly and produce similar syntax trees.
2020-01-03 12:33:11 +01:00
Ilya Biryukov
c1bbefef9d [Syntax] Use a hash table to search for tokens by their location
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.

The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
2019-12-18 12:24:00 +01:00
Ilya Biryukov
1ad15046dc [Syntax] Allow to mutate syntax trees
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.

The public interface of the syntax library now includes facilities to:
    1. perform type-safe modifications of syntax trees,
    2. compute textual replacements to apply the modifications,
    3. create syntax trees not backed by the source code.

For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.

The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.

Reviewers: sammccall, gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64573
2019-12-18 12:19:03 +01:00
Ilya Biryukov
be14a22b47 [Syntax] Build nodes for simple cases of top level declarations
Summary:
More complicated nodes (e.g. template declarations) will be implemented
in the follow-up patches.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70856
2019-12-12 08:04:22 +01:00
Ilya Biryukov
242649c17c [Syntax] Show input code on tests failures. NFC 2019-12-05 16:25:35 +01:00
Ilya Biryukov
e702bdb859 [Syntax] Build SimpleDeclaration node that groups multiple declarators
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.

See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70787
2019-11-29 11:39:45 +01:00
Ilya Biryukov
58fa50f437 [Syntax] Add nodes for most common statements
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
  - expressions are wrapped into 'ExpressionStatement' instead of being
    a subclass of statement,
  - semicolons are always consumed by the leaf expressions (return,
    expression satement, etc),
  - some clang statements are not handled yet, we wrap those into an
    UnknownStatement class, which is not present in clang.

We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63835
2019-11-06 10:56:06 +01:00
Jonas Devlieghere
2b3d49b610 [Clang] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942
2019-08-14 23:04:18 +00:00
Ilya Biryukov
bfbf6b6cab [Syntax] Do not add a node for 'eof' into the tree
Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: javed.absar, kristof.beyls, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64576

llvm-svn: 368062
2019-08-06 17:07:58 +00:00
Ilya Biryukov
51dad4196e [Syntax] Move roles into a separate enum
To align with reviewer's suggestions.

llvm-svn: 365479
2019-07-09 13:31:43 +00:00
Ilya Biryukov
9b3f38f990 Reland r365355: [Syntax] Introduce syntax trees
With a fix to a PS4 buildbot crash.

llvm-svn: 365466
2019-07-09 11:32:13 +00:00
Simon Pilgrim
7e3b22758d Revert rL365355 : [Syntax] Introduce syntax trees
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.

More to follow later:
  - Operations to mutate the syntax trees and corresponding textual
    replacements.
  - Mapping between clang AST nodes and syntax tree nodes.
  - More node types corresponding to the language constructs.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: llvm-commits, mgorny, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe

llvm-svn: 365465
2019-07-09 11:26:35 +00:00
Ilya Biryukov
b736969edd [Syntax] Introduce syntax trees
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.

More to follow later:
  - Operations to mutate the syntax trees and corresponding textual
    replacements.
  - Mapping between clang AST nodes and syntax tree nodes.
  - More node types corresponding to the language constructs.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: llvm-commits, mgorny, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D61637

llvm-svn: 365355
2019-07-08 17:25:02 +00:00