22 Commits

Author SHA1 Message Date
Marcel Hlopko
a711a3a460 [Syntax] Build mapping from AST to syntax tree nodes
Summary:
Copy of https://reviews.llvm.org/D72446, submitting with Ilya's permission.

Only used to assign roles to child nodes for now. This is more efficient
than doing range-based queries.

In the future, will be exposed in the public API of syntax trees.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76355
2020-03-23 16:22:14 +01:00
Marcel Hlopko
88bf9b3d26 [Syntax] Build template declaration nodes
Summary:
Rollforward of
https://reviews.llvm.org/rGdd12826808f9079e164b82e64b0697a077379241 after
temporarily adding -fno-delayed-template-parsing to the TreeTest.

Original summary:

> Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
>
> Handles template declaration of all kinds.
>
> Also builds template declaration nodes for specializations and explicit
> instantiations of classes.
>
> Some missing things will be addressed in the follow-up patches:
>
> * specializations of functions and variables,
> * template parameters.

Reviewers: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76418
2020-03-19 17:43:07 +01:00
Nico Weber
881f5b5a7b Revert "[Syntax] Build template declaration nodes"
This reverts commit dd12826808f9079e164b82e64b0697a077379241.
Breaks tests on Windows, see https://reviews.llvm.org/D76346#1929208
2020-03-18 12:57:55 -04:00
Marcel Hlopko
dd12826808 [Syntax] Build template declaration nodes
Summary:
Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.

Handles template declaration of all kinds.

Also builds template declaration nodes for specializations and explicit
instantiations of classes.

Some missing things will be addressed in the follow-up patches:

specializations of functions and variables,
template parameters.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76346
2020-03-18 16:16:59 +01:00
Marcel Hlopko
7d382dcd46 [Syntax] Build declarator nodes
Summary:
Copy of https://reviews.llvm.org/D72089 with Ilya's permission. See
https://reviews.llvm.org/D72089 for the first batch of comments.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76220
2020-03-16 19:13:59 +01:00
Benjamin Kramer
adcd026838 Make llvm::StringRef to std::string conversions explicit.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.
2020-01-28 23:25:25 +01:00
Ilya Biryukov
3b929fe776 [Syntax] Assert invariants on tree structure and fix a bug in mutations
Add checks for some structural invariants when building and mutating
the syntax trees.

Fix a bug failing the invariants after mutations: the parent of nodes
added into the tree was null.
2020-01-14 16:31:08 +01:00
Ilya Biryukov
04f627f6b9 [Syntax] Build spanning SimpleDecalration for classes, structs, etc
When they are free-standing, e.g. `struct X;` or `struct X {};`.
Although this complicates the common case (of free-standing class
declarations), this ensures the less common case (e.g. `struct X {} a;`)
are handled uniformly and produce similar syntax trees.
2020-01-03 12:33:11 +01:00
Ilya Biryukov
c1bbefef9d [Syntax] Use a hash table to search for tokens by their location
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.

The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
2019-12-18 12:24:00 +01:00
Ilya Biryukov
1ad15046dc [Syntax] Allow to mutate syntax trees
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.

The public interface of the syntax library now includes facilities to:
    1. perform type-safe modifications of syntax trees,
    2. compute textual replacements to apply the modifications,
    3. create syntax trees not backed by the source code.

For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.

The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.

Reviewers: sammccall, gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64573
2019-12-18 12:19:03 +01:00
Ilya Biryukov
be14a22b47 [Syntax] Build nodes for simple cases of top level declarations
Summary:
More complicated nodes (e.g. template declarations) will be implemented
in the follow-up patches.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70856
2019-12-12 08:04:22 +01:00
Ilya Biryukov
e702bdb859 [Syntax] Build SimpleDeclaration node that groups multiple declarators
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.

See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70787
2019-11-29 11:39:45 +01:00
Ilya Biryukov
302cb3bc3d [Syntax] Add a comment explaining the pointer keys in std::map<Token*, ...>. NFC 2019-11-29 09:49:24 +01:00
Ilya Biryukov
def65bb4f5 [Syntax] Remove unused parameter from TreeBuilder::markChildToken. NFC 2019-11-29 09:45:06 +01:00
Ilya Biryukov
96065cf79f [Syntax] Silence "unused function" warning in no-assert builds. NFC
A helper `isImpicitExpr` is only used inside assert.
2019-11-07 09:37:25 +01:00
Ilya Biryukov
58fa50f437 [Syntax] Add nodes for most common statements
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
  - expressions are wrapped into 'ExpressionStatement' instead of being
    a subclass of statement,
  - semicolons are always consumed by the leaf expressions (return,
    expression satement, etc),
  - some clang statements are not handled yet, we wrap those into an
    UnknownStatement class, which is not present in clang.

We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63835
2019-11-06 10:56:06 +01:00
Ilya Biryukov
bfbf6b6cab [Syntax] Do not add a node for 'eof' into the tree
Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: javed.absar, kristof.beyls, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64576

llvm-svn: 368062
2019-08-06 17:07:58 +00:00
Ilya Biryukov
51dad4196e [Syntax] Move roles into a separate enum
To align with reviewer's suggestions.

llvm-svn: 365479
2019-07-09 13:31:43 +00:00
Ilya Biryukov
9b3f38f990 Reland r365355: [Syntax] Introduce syntax trees
With a fix to a PS4 buildbot crash.

llvm-svn: 365466
2019-07-09 11:32:13 +00:00
Simon Pilgrim
7e3b22758d Revert rL365355 : [Syntax] Introduce syntax trees
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.

More to follow later:
  - Operations to mutate the syntax trees and corresponding textual
    replacements.
  - Mapping between clang AST nodes and syntax tree nodes.
  - More node types corresponding to the language constructs.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: llvm-commits, mgorny, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe

llvm-svn: 365465
2019-07-09 11:26:35 +00:00
Bill Wendling
b9816f674b Add parentheses to silence warnings.
llvm-svn: 365395
2019-07-08 22:01:07 +00:00
Ilya Biryukov
b736969edd [Syntax] Introduce syntax trees
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.

More to follow later:
  - Operations to mutate the syntax trees and corresponding textual
    replacements.
  - Mapping between clang AST nodes and syntax tree nodes.
  - More node types corresponding to the language constructs.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: llvm-commits, mgorny, cfe-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D61637

llvm-svn: 365355
2019-07-08 17:25:02 +00:00