924 Commits

Author SHA1 Message Date
Mehdi Amini
eb103104ef Revert "[Serialization] Read the initializer for interesting static variables before consuming it"
This reverts commit 11b059145d177ee287c7ada9864addf8d083c160.

The premerge bot is broken.
2024-05-14 19:53:38 -07:00
Chuanqi Xu
11b059145d [Serialization] Read the initializer for interesting static variables before consuming it
Close https://github.com/llvm/llvm-project/issues/91418

Since we load the variable's initializers lazily, it'd be problematic
if the initializers dependent on each other. So here we try to load the
initializers of static variables to make sure they are passed to code
generator by order. If we read any thing interesting, we would consume
that before emitting the current declaration.
2024-05-15 10:29:49 +08:00
Krystian Stasiowski
1202837302
Reapply "[Clang] Unify interface for accessing template arguments as written for class/variable template specializations (#81642)" (#91393)
Reapplies #81642, fixing the crash which occurs when running the lldb test suite.
2024-05-14 16:09:57 -04:00
Adrian Prantl
c6855ab24e Revert "[Clang] Unify interface for accessing template arguments as written for class/variable template specializations (#81642)"
This reverts commit 7115ed0fff027b65fa76fdfae215ed1382ed1473.

This commit broke several LLDB tests.

https://green.lab.llvm.org/job/llvm.org/view/LLDB/job/as-lldb-cmake/3480/
2024-05-07 13:03:14 -07:00
Krystian Stasiowski
7115ed0fff
[Clang] Unify interface for accessing template arguments as written for class/variable template specializations (#81642)
Our current method of storing the template arguments as written for
`(Class/Var)Template(Partial)SpecializationDecl` suffers from a number
of flaws:
- We use `TypeSourceInfo` to store `TemplateArgumentLocs` for class
template/variable template partial/explicit specializations. For
variable template specializations, this is a rather unintuitive hack (as
we store a non-type specialization as a type). Moreover, we don't ever
*need* the type as written -- in almost all cases, we only want the
template arguments (e.g. in tooling use-cases).
- The template arguments as written are stored in a number of redundant
data members. For example, `(Class/Var)TemplatePartialSpecialization`
have their own `ArgsAsWritten` member that stores an
`ASTTemplateArgumentListInfo` (the template arguments).
`VarTemplateSpecializationDecl` has yet _another_ redundant member
"`TemplateArgsInfo`" that also stores an `ASTTemplateArgumentListInfo`.

This patch eliminates all
`(Class/Var)Template(Partial)SpecializationDecl` members which store the
template arguments as written, and turns the `ExplicitInfo` member into
a `llvm::PointerUnion<const ASTTemplateArgumentListInfo*,
ExplicitInstantiationInfo*>` (to avoid unnecessary allocations when the
declaration isn't an explicit instantiation). The template arguments as
written are now accessed via `getTemplateArgsWritten` in all cases.

The "most breaking" change is to AST Matchers, insofar that `hasTypeLoc`
will no longer match class template specializations (since they no
longer store the type as written).
2024-05-07 14:45:52 -04:00
Chuanqi Xu
947b062823 Reland "[Modules] No transitive source location change (#86912)"
This relands 6c31104.

The patch was reverted due to incorrectly introduced alignment. And the
patch was re-commited after fixing the alignment issue.

Following off are the original message:

This is part of "no transitive change" patch series, "no transitive
source location change". I talked this with @Bigcheese in the tokyo's
WG21 meeting.

The idea comes from @jyknight posted on LLVM discourse. That for:

```
// A.cppm
export module A;
...

// B.cppm
export module B;
import A;
...

//--- C.cppm
export module C;
import C;
```

Almost every time A.cppm changes, we need to recompile `B`. Due to we
think the source location is significant to the semantics. But it may be
good if we can avoid recompiling `C` if the change from `A` wouldn't
change the BMI of B.

This patch only cares source locations. So let's focus on source
location's example. We can see the full example from the attached test.

```
//--- A.cppm
export module A;
export template <class T>
struct C {
    T func() {
        return T(43);
    }
};
export int funcA() {
    return 43;
}

//--- A.v1.cppm
export module A;

export template <class T>
struct C {
    T func() {
        return T(43);
    }
};
export int funcA() {
    return 43;
}

//--- B.cppm
export module B;
import A;

export int funcB() {
    return funcA();
}

//--- C.cppm
export module C;
import A;
export void testD() {
    C<int> c;
    c.func();
}
```

Here the only difference between `A.cppm` and `A.v1.cppm` is that
`A.v1.cppm` has an additional blank line. Then the test shows that two
BMI of `B.cppm`, one specified `-fmodule-file=A=A.pcm` and the other
specified `-fmodule-file=A=A.v1.pcm`, should have the bit-wise same
contents.

However, it is a different story for C, since C instantiates templates
from A, and the instantiation records the source information from module
A, which is different from `A` and `A.v1`, so it is expected that the
BMI `C.pcm` and `C.v1.pcm` can and should differ.

To fully understand the patch, we need to understand how we encodes
source locations and how we serialize and deserialize them.

For source locations, we encoded them as:

```
|
|
| _____ base offset of an imported module
|
|
|
|_____ base offset of another imported module
|
|
|
|
| ___ 0
```

As the diagram shows, we encode the local (unloaded) source location
from 0 to higher bits. And we allocate the space for source locations
from the loaded modules from high bits to 0. Then the source locations
from the loaded modules will be mapped to our source location space
according to the allocated offset.

For example, for,

```
// a.cppm
export module a;
...

// b.cppm
export module b;
import a;
...
```

Assuming the offset of a source location (let's name the location as
`S`) in a.cppm is 45 and we will record the value `45` into the BMI
`a.pcm`. Then in b.cppm, when we import a, the source manager will
allocate a space for module 'a' (according to the recorded number of
source locations) as the base offset of module 'a' in the current source
location spaces. Let's assume the allocated base offset as 90 in this
example. Then when we want to get the location in the current source
location space for `S`, we can get it simply by adding `45` to `90` to
`135`. Finally we can get the source location for `S` in module B as
`135`.

And when we want to write module `b`, we would also write the source
location of `S` as `135` directly in the BMI. And to clarify the
location `S` comes from module `a`, we also need to record the base
offset of module `a`, 90 in the BMI of `b`.

Then the problem comes. Since the base offset of module 'a' is computed
by the number source locations in module 'a'. In module 'b', the
recorded base offset of module 'a' will change every time the number of
source locations in module 'a' increase or decrease. In other words, the
contents of BMI of B will change every time the number of locations in
module 'a' changes. This is pretty sensitive. Almost every change will
change the number of locations. So this is the problem this patch want
to solve.

Let's continue with the existing design to understand what's going on.
Another interesting case is:

```
// c.cppm
export module c;
import whatever;
import a;
import b;
...
```

In `c.cppm`, when we import `a`, we still need to allocate a base
location offset for it, let's say the value becomes to `200` somehow.
Then when we reach the location `S` recorded in module `b`, we need to
translate it into the current source location space. The solution is
quite simple, we can get it by `135 + (200 - 90) = 245`. In another
word, the offset of a source location in current module can be computed
as `Recorded Offset + Base Offset of the its module file - Recorded Base
Offset`.

Then we're almost done about how we handle the offset of source
locations in serializers.

From the abstract level, what we want to do is to remove the hardcoded
base offset of imported modules and remain the ability to calculate the
source location in a new module unit. To achieve this, we need to be
able to find the module file owning a source location from the encoding
of the source location.

So in this patch, for each source location, we will store the local
offset of the location and the module file index. For the above example,
in `b.pcm`, the source location of `S` will be recorded as `135`
directly. And in the new design, the source location of `S` will be
recorded as `<1, 45>`. Here `1` stands for the module file index of `a`
in module `b`. And `45` means the offset of `S` to the base offset of
module `a`.

So the trade-off here is that, to make the BMI more independent, we need
to record more abstract information. And I feel it is worthy. The
recompilation problem of modules is really annoying and there are still
people complaining this. But if we can make this (including stopping
other changes transitively), I think this may be a killer feature for
modules. And from @Bigcheese , this should be helpful for clang explicit
modules too.

And the benchmarking side, I tested this patch against
https://github.com/alibaba/async_simple/tree/CXX20Modules. No
significant change on compilation time. The size of .pcm files becomes
to 204M from 200M. I think the trade-off is pretty fair.

I didn't use another slot to record the module file index. I tried to
use the higher 32 bits of the existing source location encodings to
store that information. This design may be safe. Since we use `unsigned`
to store source locations but we use uint64_t in serialization. And
generally `unsigned` is 32 bit width in most platforms. So it might not
be a safe problem. Since all the bits we used to store the module file
index is not used before. So the new encodings may be:

```
   |-----------------------|-----------------------|
   |           A           |         B         | C |

  * A: 32 bit. The index of the module file in the module manager + 1.
  * The +1
          here is necessary since we wish 0 stands for the current
module file.
  * B: 31 bit. The offset of the source location to the module file
  * containing it.
  * C: The macro bit. We rotate it to the lowest bit so that we can save
  * some
          space in case the index of the module file is 0.
```

(The B and C is the existing raw encoding for source locations)

Another reason to reuse the same slot of the source location is to
reduce the impact of the patch. Since there are a lot of places assuming
we can store and get a source location from a slot. And if I tried to
add another slot, a lot of codes breaks. I don't feel it is worhty.

Another impact of this decision is that, the existing small
optimizations for encoding source location may be invalided. The key of
the optimization is that we can turn large values into small values then
we can use VBR6 format to reduce the size. But if we decided to put the
module file index into the higher bits, then maybe it simply doesn't
work. An example may be the `SourceLocationSequence` optimization.

This will only affect the size of on-disk .pcm files. I don't expect
this impact the speed and memory use of compilations. And seeing my
small experiments above, I feel this trade off is worthy.

The mental model for handling source location offsets is not so complex
and I believe we can solve it by adding module file index to each stored
source location.

For the practical side, since the source location is pretty sensitive,
and the patch can pass all the in-tree tests and a small scale projects,
I feel it should be correct.

I'll continue to work on no transitive decl change and no transitive
identifier change (if matters) to achieve the goal to stop the
propagation of unnecessary changes. But all of this depends on this
patch. Since, clearly, the source locations are the most sensitive
thing.

---

The release nots and documentation will be added seperately.
2024-05-06 13:35:16 +08:00
Volodymyr Sapsai
6d7d8e5b59
[Modules] Detect ODR mismatches for enums in non-C++ like in C++. (#90298)
There is no reason for C and Objective-C to differ from C++ in this
matter.

rdar://85531830
2024-05-02 18:10:36 -07:00
Chuanqi Xu
d333a0de68 Revert "[Modules] No transitive source location change (#86912)"
This reverts commit 6c3110464bac3600685af9650269b0b2b8669d34.

Required by the post commit comments: https://github.com/llvm/llvm-project/pull/86912
2024-04-30 22:32:02 +08:00
Chuanqi Xu
6c3110464b
[Modules] No transitive source location change (#86912)
This is part of "no transitive change" patch series, "no transitive
source location change". I talked this with @Bigcheese in the tokyo's
WG21 meeting.

The idea comes from @jyknight posted on LLVM discourse. That for:

```
// A.cppm
export module A;
...

// B.cppm
export module B;
import A;
...

//--- C.cppm
export module C;
import C;
```

Almost every time A.cppm changes, we need to recompile `B`. Due to we
think the source location is significant to the semantics. But it may be
good if we can avoid recompiling `C` if the change from `A` wouldn't
change the BMI of B.

# Motivation Example

This patch only cares source locations. So let's focus on source
location's example. We can see the full example from the attached test.

```
//--- A.cppm
export module A;
export template <class T>
struct C {
    T func() {
        return T(43);
    }
};
export int funcA() {
    return 43;
}

//--- A.v1.cppm
export module A;

export template <class T>
struct C {
    T func() {
        return T(43);
    }
};
export int funcA() {
    return 43;
}

//--- B.cppm
export module B;
import A;

export int funcB() {
    return funcA();
}

//--- C.cppm
export module C;
import A;
export void testD() {
    C<int> c;
    c.func();
}
```

Here the only difference between `A.cppm` and `A.v1.cppm` is that
`A.v1.cppm` has an additional blank line. Then the test shows that two
BMI of `B.cppm`, one specified `-fmodule-file=A=A.pcm` and the other
specified `-fmodule-file=A=A.v1.pcm`, should have the bit-wise same
contents.

However, it is a different story for C, since C instantiates templates
from A, and the instantiation records the source information from module
A, which is different from `A` and `A.v1`, so it is expected that the
BMI `C.pcm` and `C.v1.pcm` can and should differ.

# Internal perspective of status quo

To fully understand the patch, we need to understand how we encodes
source locations and how we serialize and deserialize them.

For source locations, we encoded them as:

```
|
|
| _____ base offset of an imported module
|
|
|
|_____ base offset of another imported module
|
|
|
|
| ___ 0
```

As the diagram shows, we encode the local (unloaded) source location
from 0 to higher bits. And we allocate the space for source locations
from the loaded modules from high bits to 0. Then the source locations
from the loaded modules will be mapped to our source location space
according to the allocated offset.

For example, for,

```
// a.cppm
export module a;
...

// b.cppm
export module b;
import a;
...
```

Assuming the offset of a source location (let's name the location as
`S`) in a.cppm is 45 and we will record the value `45` into the BMI
`a.pcm`. Then in b.cppm, when we import a, the source manager will
allocate a space for module 'a' (according to the recorded number of
source locations) as the base offset of module 'a' in the current source
location spaces. Let's assume the allocated base offset as 90 in this
example. Then when we want to get the location in the current source
location space for `S`, we can get it simply by adding `45` to `90` to
`135`. Finally we can get the source location for `S` in module B as
`135`.

And when we want to write module `b`, we would also write the source
location of `S` as `135` directly in the BMI. And to clarify the
location `S` comes from module `a`, we also need to record the base
offset of module `a`, 90 in the BMI of `b`.

Then the problem comes. Since the base offset of module 'a' is computed
by the number source locations in module 'a'. In module 'b', the
recorded base offset of module 'a' will change every time the number of
source locations in module 'a' increase or decrease. In other words, the
contents of BMI of B will change every time the number of locations in
module 'a' changes. This is pretty sensitive. Almost every change will
change the number of locations. So this is the problem this patch want
to solve.

Let's continue with the existing design to understand what's going on.
Another interesting case is:

```
// c.cppm
export module c;
import whatever;
import a;
import b;
...
```

In `c.cppm`, when we import `a`, we still need to allocate a base
location offset for it, let's say the value becomes to `200` somehow.
Then when we reach the location `S` recorded in module `b`, we need to
translate it into the current source location space. The solution is
quite simple, we can get it by `135 + (200 - 90) = 245`. In another
word, the offset of a source location in current module can be computed
as `Recorded Offset + Base Offset of the its module file - Recorded Base
Offset`.

Then we're almost done about how we handle the offset of source
locations in serializers.

# The high level design of current patch

From the abstract level, what we want to do is to remove the hardcoded
base offset of imported modules and remain the ability to calculate the
source location in a new module unit. To achieve this, we need to be
able to find the module file owning a source location from the encoding
of the source location.

So in this patch, for each source location, we will store the local
offset of the location and the module file index. For the above example,
in `b.pcm`, the source location of `S` will be recorded as `135`
directly. And in the new design, the source location of `S` will be
recorded as `<1, 45>`. Here `1` stands for the module file index of `a`
in module `b`. And `45` means the offset of `S` to the base offset of
module `a`.

So the trade-off here is that, to make the BMI more independent, we need
to record more abstract information. And I feel it is worthy. The
recompilation problem of modules is really annoying and there are still
people complaining this. But if we can make this (including stopping
other changes transitively), I think this may be a killer feature for
modules. And from @Bigcheese , this should be helpful for clang explicit
modules too.

And the benchmarking side, I tested this patch against
https://github.com/alibaba/async_simple/tree/CXX20Modules. No
significant change on compilation time. The size of .pcm files becomes
to 204M from 200M. I think the trade-off is pretty fair.

# Some low level details

I didn't use another slot to record the module file index. I tried to
use the higher 32 bits of the existing source location encodings to
store that information. This design may be safe. Since we use `unsigned`
to store source locations but we use uint64_t in serialization. And
generally `unsigned` is 32 bit width in most platforms. So it might not
be a safe problem. Since all the bits we used to store the module file
index is not used before. So the new encodings may be:

```
   |-----------------------|-----------------------|
   |           A           |         B         | C |

  * A: 32 bit. The index of the module file in the module manager + 1. The +1
          here is necessary since we wish 0 stands for the current module file.
  * B: 31 bit. The offset of the source location to the module file containing it.
  * C: The macro bit. We rotate it to the lowest bit so that we can save some 
          space in case the index of the module file is 0.
```

(The B and C is the existing raw encoding for source locations)

Another reason to reuse the same slot of the source location is to
reduce the impact of the patch. Since there are a lot of places assuming
we can store and get a source location from a slot. And if I tried to
add another slot, a lot of codes breaks. I don't feel it is worhty.

Another impact of this decision is that, the existing small
optimizations for encoding source location may be invalided. The key of
the optimization is that we can turn large values into small values then
we can use VBR6 format to reduce the size. But if we decided to put the
module file index into the higher bits, then maybe it simply doesn't
work. An example may be the `SourceLocationSequence` optimization.

This will only affect the size of on-disk .pcm files. I don't expect
this impact the speed and memory use of compilations. And seeing my
small experiments above, I feel this trade off is worthy.

# Correctness

The mental model for handling source location offsets is not so complex
and I believe we can solve it by adding module file index to each stored
source location.

For the practical side, since the source location is pretty sensitive,
and the patch can pass all the in-tree tests and a small scale projects,
I feel it should be correct.

# Future Plans

I'll continue to work on no transitive decl change and no transitive
identifier change (if matters) to achieve the goal to stop the
propagation of unnecessary changes. But all of this depends on this
patch. Since, clearly, the source locations are the most sensitive
thing.

---

The release nots and documentation will be added seperately.
2024-04-30 15:57:58 +08:00
Chuanqi Xu
d86cc73bbf [NFC] [Serialization] Avoid using DeclID directly as much as possible
This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is converted to
the use of LocalDeclID or GlobalDeclID. This is helpful to increase the
readability and type safety.
2024-04-25 14:59:09 +08:00
Chuanqi Xu
72b58146b1 Revert "[NFC] [Serialization] Avoid using DeclID directly as much as possible"
This reverts commit 42070a5c092ed420bf92ebf38229c594885e94c7.

I forgot to touch lldb.
2024-04-25 14:26:07 +08:00
Chuanqi Xu
42070a5c09 [NFC] [Serialization] Avoid using DeclID directly as much as possible
This patch tries to remove all the direct use of DeclID except the real
low level reading and writing. All the use of DeclID is converted to
the use of LocalDeclID or GlobalDeclID. This is helpful to increase the
readability and type safety.
2024-04-25 14:14:05 +08:00
Chuanqi Xu
c2a98fdeb3
[NFC] Move DeclID from serialization/ASTBitCodes.h to AST/DeclID.h (#89873)
Previously, the DeclID is defined in serialization/ASTBitCodes.h under
clang::serialization namespace. However, actually the DeclID is not
purely used in serialization part. The DeclID is already widely used in
AST and all around the clang project via classes like `LazyPtrDecl` or
calling `ExternalASTSource::getExernalDecl()`. All such uses are via the
raw underlying type of `DeclID` as `uint32_t`. This is not pretty good.

This patch moves the DeclID class family to a new header `AST/DeclID.h`
so that the whole project can use the wrapped class `DeclID`,
`GlobalDeclID` and `LocalDeclID` instead of the raw underlying type.
This can improve the readability and the type safety.
2024-04-25 13:53:22 +08:00
Chuanqi Xu
b467c6b536 [NFC] [Serialization] Turn type alias GlobalDeclID into a class
Succsessor of b8e3b2ad66cf78ad2b. This patch also converts the type
alias GlobalDeclID to a class to improve the readability and type
safety.
2024-04-23 17:52:58 +08:00
Chuanqi Xu
b8e3b2ad66 [NFC] [Serialization] Turn type alias LocalDeclID into class
Previously, the LocalDeclID and GlobalDeclID are defined as:

```
using LocalDeclID = DeclID;
using GlobalDeclID = DeclID;
```

This is more or less concerning that we may misuse LocalDeclID and
GlobalDeclID without understanding it. There is also a FIXME saying
this.

This patch tries to turn LocalDeclID into a class to improve the type
safety here.
2024-04-23 16:56:14 +08:00
Chuanqi Xu
07b1177eed [NFC] [Serialization] Use semantical type DeclID instead of raw type 'uint32_t'
This patch tries to use DeclID in the code bases to avoid use the raw
type 'uint32_t'. It is problematic to use the raw type 'uint32_t' if we
want to change the type of DeclID some day.
2024-04-23 12:44:00 +08:00
Chuanqi Xu
d26dd58ca5 [StmtProfile] Don't profile the body of lambda expressions
Close https://github.com/llvm/llvm-project/issues/87609

We tried to profile the body of the lambda expressions in
https://reviews.llvm.org/D153957. But as the original comments show,
it is indeed dangerous. After we tried to skip calculating the ODR
hash values recently, we have fall into this trap twice.

So in this patch, I choose to not profile the body of the lambda
expression. The signature of the lambda is still profiled.
2024-04-16 15:41:26 +08:00
Sirraide
ef164cee90
[Clang] [C++26] Implement P2573R2: = delete("should have a reason"); (#86526)
This implements support for the `= delete("message")` syntax that was
only just added to C++26
([P2573R2](https://isocpp.org/files/papers/P2573R2.html#proposal-scope)).
2024-04-14 12:30:01 +02:00
Nikolas Klauser
ed06b847d4
Revert "[Clang] Reduce the size of Decl and classes derived from it" (#88654)
Reverts llvm/llvm-project#87361

On 32 bit platforms there is only a single bit available in the
`DeclCtx`, resulting in an assertion failure.
2024-04-14 12:25:49 +02:00
Nikolas Klauser
c6f9c84e49
[Clang] Reduce the size of Decl and classes derived from it (#87361)
Class | Old size (in bytes) | New size (in bytes)

----------------------------------|---------------------|--------------------
Decl                              | 40                  | 32
AccessSpecDecl                    | 40                  | 40
BlockDecl                         | 128                 | 120
CapturedDecl                      | 88                  | 80
EmptyDecl                         | 40                  | 32
ExportDecl                        | 80                  | 72
ExternCContextDecl                | 72                  | 64
FileScopeAsmDecl                  | 56                  | 48
FriendDecl                        | 64                  | 56
FriendTemplateDecl                | 64                  | 64
ImplicitConceptSpecializationDecl | 40                  | 40
ImportDecl                        | 56                  | 48
LifetimeExtendedTemporaryDecl     | 72                  | 64
LinkageSpecDecl                   | 80                  | 72
NamedDecl                         | 48                  | 40
ObjCPropertyImplDecl              | 96                  | 88
PragmaCommentDecl                 | 40                  | 40
PragmaDetectMismatchDecl          | 48                  | 40
RequiresExprBodyDecl              | 72                  | 64
StaticAssertDecl                  | 64                  | 56
TopLevelStmtDecl                  | 88                  | 80
TranslationUnitDecl               | 104                 | 96
BaseUsingDecl                     | 56                  | 48
UsingDecl                         | 88                  | 80
UsingEnumDecl                     | 72                  | 64
HLSLBufferDecl                    | 96                  | 88
LabelDecl                         | 80                  | 72
NamespaceAliasDecl                | 96                  | 88
NamespaceDecl                     | 112                 | 104
ObjCCompatibleAliasDecl           | 56                  | 48
ObjCContainerDecl                 | 88                  | 80
ObjCMethodDecl                    | 136                 | 128
ObjCPropertyDecl                  | 128                 | 120
TemplateDecl                      | 64                  | 56
BuiltinTemplateDecl               | 72                  | 64
TypeDecl                          | 64                  | 56
UnresolvedUsingIfExistsDecl       | 48                  | 40
UsingDirectiveDecl                | 88                  | 80
UsingPackDecl                     | 64                  | 56
UsingShadowDecl                   | 80                  | 72
ValueDecl                         | 56                  | 48

When parsing libc++'s `<string>` header the used memory is reduced from
42.8MB to 42.5MB.
2024-04-14 12:08:30 +02:00
Chuanqi Xu
f21ead0675
[C++20] [Modules] [Reduced BMI] Remove unreachable decls GMF in redued BMI (#88359)
Following of https://github.com/llvm/llvm-project/pull/76930

This follows the idea of "only writes what we writes", which I think is
the most natural and efficient way to implement this optimization.

We start writing the BMI from the first declaration in module purview
instead of the global module fragment, so that everything in the GMF
untouched won't be written in the BMI naturally.

The exception is, as I said in
https://github.com/llvm/llvm-project/pull/76930, when we write a
declaration we need to write its decl context, and when we write the
decl context, we need to write everything from it. So when we see
`std::vector`, we basically need to write everything under namespace
std. This violates our intention. To fix this, this patch delays the
writing of namespace in the GMF.

From my local measurement, the size of the BMI decrease to 90M from 112M
for a local modules build. I think this is significant.

This feature will be covered under the experimental reduced BMI so that
it won't affect any existing users. So I'd like to land this when the CI
gets green.

Documents will be added seperately.
2024-04-12 12:51:58 +08:00
Krystian Stasiowski
4e6d18f406
[Clang][AST] Track whether template template parameters used the 'typename' keyword (#88139)
This patch adds a `Typename` bit-field to `TemplateTemplateParmDecl`
which stores whether the template template parameter was declared with
the `typename` keyword.
2024-04-11 13:20:05 -04:00
Chuanqi Xu
75edf0c18c [NFC] [Serialization] Avoid accessing PendingBodies as much as possible
The 'HaveBody' parameter in isConsumerInterestedIn is only used for the function decl if it
doesn't have a body already. It should be relatively less frequent than
the call to isConsumerInterestedIn. So we can delay the computing of
`HaveBdoy` to make it more efficient.
2024-04-11 11:08:58 +08:00
Chuanqi Xu
45aec9a0b5 [NFC] [Serialization] Remove redundant hasPendingBody member
The hasPendingBody member is redundant with the
PendingBodies.count(Decl*) method. This patch removes the redundant
hasPendingBody member and the corresponding InterestingDecl struct.
2024-04-09 17:50:23 +08:00
Chuanqi Xu
3f6bc1adf8 [C++20] [Moduls] Avoid computing odr hash for functions from comparing constraint expression
Previously we disabled to compute ODR hash for declarations from the
global module fragment. However, we missed the case that the functions
lives in the concept requiments (see the attached the test files for
example). And the mismatch causes the potential crashment.

Due to we will set the function body as lazy after we deserialize it and
we will only take its body when needed. However, we don't allow to take
the body during deserializing. So it is actually potentially problematic
if we set the body as lazy first and computing the hash value of the
function, which requires to deserialize its body. So we will meet a
crash here.

This patch tries to solve the issue by not taking the body of the
function from GMF. Note that we can't skip comparing the constraint
expression from the GMF directly since it is an key part of the
function selecting and it may be the reason why we can't return 0
directly for `FunctionDecl::getODRHash()` from the GMF.
2024-03-11 11:39:21 +08:00
Chuanqi Xu
49775b1dc0
[Serialization] Record whether the ODR is skipped (#82302)
Close https://github.com/llvm/llvm-project/issues/80570.

In

a0b6747804,
we skipped ODR checks for decls in GMF. Then it should be natural to
skip storing the ODR values in BMI.

Generally it should be fine as long as the writer and the reader keep
consistent.

However, the use of preamble in clangd shows the tricky part.

For,

```
// test.cpp
module;

// any one off these is enough to crash clangd
// #include <iostream>
// #include <string_view>
// #include <cmath>
// #include <system_error>
// #include <new>
// #include <bit>
// probably many more

// only ok with libc++, not the system provided libstdc++ 13.2.1

// these are ok

export module test;
```

clangd will store the headers as preamble to speedup the parsing and the
preamble reuses the serialization techniques. (Generally we'd call the
preamble as PCH. However it is not true strictly. I've tested the PCH
wouldn't be problematic.) However, the tricky part is that the preamble
is not modules. It literally serialiaze and deserialize things. So
before clangd parsing the above test module, clangd will serialize the
headers into the preamble. Note that there is no concept like GMF now.
So the ODR bits are stored. However, when clangd parse the file
actually, the decls from preamble are thought as in GMF literally, then
hte ODR bits are skipped. Then mismatch happens.

To solve the problem, this patch adds another bit for decls to record
whether or not the ODR bits are skipped.
2024-02-20 13:31:28 +08:00
Chuanqi Xu
8eea582dcb
[C++20] [Modules] Introduce -fskip-odr-check-in-gmf (#79959)
Close https://github.com/llvm/llvm-project/issues/79240

Cite the comment from @mizvekov in
//github.com/llvm/llvm-project/issues/79240:

> There are two kinds of bugs / issues relevant here:
>
> Clang bugs that this change hides
> Here we can add a Frontend flag that disables the GMF ODR check, just
> so
> we can keep tracking, testing and fixing these issues.
> The Driver would just always pass that flag.
> We could add that flag in this current issue.
> Bugs in user code:
> I don't think it's worth adding a corresponding Driver flag for
> controlling the above Frontend flag, since we intend it's behavior to
> become default as we fix the problems, and users interested in testing
> the more strict behavior can just use the Frontend flag directly.

This patch follows the suggestion:
- Introduce the CC1 flag `-fskip-odr-check-in-gmf` which is by default
off, so that the every existing test will still be tested with checking
ODR violations.
- Passing `-fskip-odr-check-in-gmf` in the driver to keep the behavior
we intended.
- Edit the document to tell the users who are still interested in more
strict checks can use `-Xclang -fno-skip-odr-check-in-gmf` to get the
existing behavior.
2024-02-01 13:44:32 +08:00
Chuanqi Xu
a0b6747804 [C++20] [Modules] Don't perform ODR checks in GMF
Close https://github.com/llvm/llvm-project/issues/79240.

See the linked issue for details. Given the frequency of issue reporting
about false positive ODR checks (I received private issue reports too),
I'd like to backport this to 18.x too.
2024-01-29 11:44:59 +08:00
Chuanqi Xu
dd3e6c87f3
Support C++20 Modules in clang-repl (#79261)
This comes from when I playing around clang-repl with moduels : )

I succeeded to import std with https://libcxx.llvm.org/Modules.html and
calling `std::printf` after this patch.

I want to put the documentation part to
https://clang.llvm.org/docs/StandardCPlusPlusModules.html in a separate
commit.
2024-01-24 15:45:05 +08:00
cor3ntin
e90e43fb9c
[Clang][NFC] Rename CXXMethodDecl::isPure -> is VirtualPure (#78463)
To avoid any possible confusion with the notion of pure function and the
gnu::pure attribute.
2024-01-18 15:30:58 +01:00
Egor Zhdan
77d21e758e
[APINotes] Upstream dependencies of Sema logic to apply API Notes to decls
This upstreams more of the Clang API Notes functionality that is
currently implemented in the Apple fork:
https://github.com/apple/llvm-project/tree/next/clang/lib/APINotes

This is the largest chunk of the API Notes functionality in the
upstreaming process. I will soon submit a follow-up patch to actually
enable usage of this functionality by having a Clang driver flag that
enables API Notes, along with tests.
2024-01-17 13:13:10 +00:00
Craig Topper
142f270c27 Recommit "[AST] Use APIntStorage to fix memory leak in EnumConstantDecl. (#78311)"
With lldb build fix.

Original message:

EnumConstantDecl is allocated by the ASTContext allocator so the
destructor is never called.

This patch takes a similar approach to IntegerLiteral by using
APIntStorage to allocate large APSInts using the ASTContext allocator as
well.

The downside is that an additional heap allocation and copy of the data
needs to be made when calling getInitValue if the APSInt is large.

Fixes #78160.
2024-01-16 13:52:17 -08:00
Craig Topper
f3d534c425 Revert "[AST] Use APIntStorage to fix memory leak in EnumConstantDecl. (#78311)"
This reverts commit 4737959d91fab7673b1bb642f88658bb2a24d723.

Missed an lldb update.
2024-01-16 12:39:47 -08:00
Craig Topper
4737959d91
[AST] Use APIntStorage to fix memory leak in EnumConstantDecl. (#78311)
EnumConstantDecl is allocated by the ASTContext allocator so the
destructor is never called.

This patch takes a similar approach to IntegerLiteral by using
APIntStorage to allocate large APSInts using the ASTContext allocator as
well.

The downside is that an additional heap allocation and copy of the data
needs to be made when calling getInitValue if the APSInt is large.

Fixes #78160.
2024-01-16 12:10:38 -08:00
Chuanqi Xu
52770d83bf [Serialization] Don't pack bits for the function scope index of ParmVarDecl
Close https://github.com/llvm/llvm-project/issues/76443

Previously we assume the bits of the function scope index of ParmVarDecl
won't exceed 8. But this is a misreading. See the implementation of
`ParmVarDecl::getParameterIndex()`, which may
exceed the size of the normal bitfield. So it may be better to not
pack these bits.
2023-12-28 11:04:11 +08:00
Chuanqi Xu
2203a4e6e0 [NFC] [Serialization] Improve AST serialization by reordering packed
bits and extract big bits from packed bits

Previously I tried to improve the size of .pcm files by introducing
packed bits. And I find we can improve it further by reordering the
bits.

The secret comes from the VBR format. We can find the formal definition
of VBR format in the doc of LLVM. The VBR format will be pretty
efficicent for small numbers.

For example, if we need to pack 8 bits into a value and the stored value
is 0xf0, the actual stored value will be 0b000111'110000, which takes 12
bits actually. However, if we changed the order to be 0x0f, then we
can store it as 0b001111, which takes 6 bits only now.

So we can improve the size by placing bits with lower probability to be
1 in the higher bits and extract bit bigs from the packed bits to make
it possible to be optimized by VBR.

After this patch, the size of std module becomes to 27.7MB from 28.1MB.
2023-12-21 16:35:20 +08:00
Chuanqi Xu
565e5e861f Recommit [NFC] [Serialization] Packing more bits and refactor AbbrevToUse
This patch tries to pack more bits into a value to reduce the size of
.pcm files. Also, after we introduced BitsPackers, it may slightly
better to adjust the way we use Abbrev.

After this patch, the size of the BMI for std module reduce from 28.94MB
to 28.1 MB.

This was reverted due to it broke the build of lldb. The reason that we
skip the serialization of a source location incorrectly. And this patch
now fixes that.
2023-12-21 10:30:12 +08:00
Augusto Noronha
eccc1cca71 Revert "[NFC] [Serialization] Packing more bits and refactor AbbrevToUse"
This reverts commit 9cdb825a4f1bf9e75829d03879620c6144d0b7bc.
2023-12-15 14:43:25 -08:00
Chuanqi Xu
9cdb825a4f [NFC] [Serialization] Packing more bits and refactor AbbrevToUse
This patch tries to pack more bits into a value to reduce the size of
.pcm files. Also, after we introduced BitsPackers, it may slightly
better to adjust the way we use Abbrev.

After this patch, the size of the BMI for std module reduce from 28.94MB
to 28.1 MB.
2023-12-15 11:12:52 +08:00
Chuanqi Xu
65b12a8af3 Recommit [NFC] [Serialization] Packing more bits
This patch tries to reduce the size of the BMIs by packing more bits
into an unsigned integer.

This patch was reverted due to buildbot failure report. But it should be
irrevelent after I took a double look. So I tried to recommit this NFC
change again.
2023-12-11 16:47:51 +08:00
Chuanqi Xu
8c33462781 Revert "[NFC] [Serialization] Packing more bits"
This reverts commit 9406ea3fe32e59a7d28de0dcbd0317b4cdfa4c62.

There are build bots complaining this. Revert it first to try to keep
the bots green.
2023-12-11 14:06:32 +08:00
Chuanqi Xu
9406ea3fe3 [NFC] [Serialization] Packing more bits
This patch tries to reduce the size of the BMIs by packing more bits
into an unsigned integer.
2023-12-11 10:18:12 +08:00
Chuanqi Xu
649e8111a9 [C++20] [Modules] Handling capturing strucuted bindings
Close https://github.com/llvm/llvm-project/issues/72828.

This should be an overlook that we extend the type of captures but we
forgot to fix it in deserializer side.
2023-11-29 11:45:31 +08:00
Vlad Serebrennikov
edd690b02e
[clang][NFC] Refactor TagTypeKind (#71160)
This patch converts TagTypeKind into scoped enum. Among other benefits,
this allows us to forward-declare it where necessary.
2023-11-03 21:45:39 +04:00
Chuanqi Xu
48be81e172
[NFC] [Serializer] Pack information in serializer (#69287)
Previously, the boolean values will occupy spaces that can contain
integers. It wastes the spaces especially if the boolean values are
serialized consecutively. The patch tries to pack such consecutive
boolean values (and enum values) so that we can save more spaces and so
the times.

Before the patch, we need 4.478s (in my machine) to build the std module
(https://libcxx.llvm.org/Modules.html) with 28712 bytes for size of the
BMI. After the patch, the time becomes to 4.374s and the size becomes to
27388 bytes for the size of the BMI.

This is intended to be a NFC patch.

This patch doesn't optimize all such cases. We can do it later after we
have consensus on this.
2023-11-03 21:59:44 +08:00
Vlad Serebrennikov
8775947633
[clang][NFC] Refactor clang::Linkage (#71049)
This patch introduces a new enumerator `Invalid = 0`, shifting other enumerators by +1. Contrary to how it might sound, this actually affirms status quo of how this enum is stored in `clang::Decl`:
```
  /// If 0, we have not computed the linkage of this declaration.
  /// Otherwise, it is the linkage + 1.
  mutable unsigned CacheValidAndLinkage : 3;
```
This patch makes debuggers to not be mistaken about enumerator stored in this bit-field. It also converts `clang::Linkage` to a scoped enum.
2023-11-02 20:57:29 +04:00
Vlad Serebrennikov
a8ead56068
[clang][NFC] Rename ArgPassingKind to RecordArgPassingKind (#70955)
During the recent refactoring (b120fe8d3288c4dca1b5427ca34839ce8833f71c) this enum was moved out of `RecordDecl`. During post-commit review it was found out that its association with `RecordDecl` should be expressed in the name.
2023-11-01 20:38:28 +04:00
Vlad Serebrennikov
65761200ce [clang][NFC] Refactor LinkageSpecDecl::LanguageIDs
This patch converts `LinkageSpecDecl::LanguageIDs` into scoped enum, and moves it to namespace scope, so that it can be forward-declared where required.
2023-11-01 16:44:34 +03:00
Vlad Serebrennikov
aaba3761db [clang][NFC] Refactor ObjCMethodDecl::ImplementationControl
This patch moves `ObjCMethodDecl::ImplementationControl` to a DeclBase.h so that it's complete at the point where corresponsing bit-field is declared. This patch also converts it to a scoped enum `clang::ObjCImplementationControl`.
2023-11-01 13:40:11 +03:00
Vlad Serebrennikov
50dec541f3 [clang][NFC] Refactor OMPDeclareReductionDecl::InitKind
This patch moves `OMPDeclareReductionDecl::InitKind` to DeclBase.h, so that it's complete at the point where corresponding bit-field is declared. This patch also converts it to scoped enum named `OMPDeclareReductionInitKind`
2023-11-01 12:40:13 +03:00