Add remark format 'Auto', which performs automatic detection of the
remark format using the magic numbers at the beginning of the remarks
files.
The RemarkLinker already did something similar, so we streamlined this
and exposed this to llvm-remarkutil.
Background: The yaml-strtab format looks just like the yaml format,
except that the values in the key/value pairs of the remarks are
deduplicated and replaced by indices into a string table (see removed
test cases for examples). The motivation behind this format was to
reduce size of the remarks files. However, it was quickly superseded by
the bitstream format.
Therefore, remove the yaml-strtab format, as it doesn't have a good
usecase anymore:
- It isn't particularly efficient
- It isn't human-readable
- It isn't straightforward to parse in external tools that can't use the
remarks library. We don't even support it in opt-viewer.
llvm-remarkutil is also missing options to parse/convert yaml-strtab, so
the chance that anyone is actually using this format is low.
Previously if a string is in the format
```
|
val
val2
val3
```
Yaml parser will error out without parsing the string. The mentioned pattern is a valid yaml str and should be parsed.
Differential Revision: https://reviews.llvm.org/D153760
At the moment, dsymutil drops all remarks without debug location.
There are many cases where debug location may be missing for remarks,
mostly due LLVM not preserving debug locations. When using bitstream
remarks for statistical analysis, those missed remarks mean we get an
incomplete picture.
The patch flips the default to keeping all remarks and leaving it to
tools that display remarks to filter out remarks without debug locations
as needed.
The new --remarks-drop-without-debug flag can be used to drop remarks
without debug locations, i.e. restore the previous behavior.
Reviewed By: thegameg
Differential Revision: https://reviews.llvm.org/D151089
Use deduction guides instead of helper functions.
The only non-automatic changes have been:
1. ArrayRef(some_uint8_pointer, 0) needs to be changed into ArrayRef(some_uint8_pointer, (size_t)0) to avoid an ambiguous call with ArrayRef((uint8_t*), (uint8_t*))
2. CVSymbol sym(makeArrayRef(symStorage)); needed to be rewritten as CVSymbol sym{ArrayRef(symStorage)}; otherwise the compiler is confused and thinks we have a (bad) function prototype. There was a few similar situation across the codebase.
3. ADL doesn't seem to work the same for deduction-guides and functions, so at some point the llvm namespace must be explicitly stated.
4. The "reference mode" of makeArrayRef(ArrayRef<T> &) that acts as no-op is not supported (a constructor cannot achieve that).
Per reviewers' comment, some useless makeArrayRef have been removed in the process.
This is a follow-up to https://reviews.llvm.org/D140896 that introduced
the deduction guides.
Differential Revision: https://reviews.llvm.org/D140955
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
Remarks are usually emitted per-TU, and for generating a standalone
remark file that can be shipped with the linked binary we need some kind
of tool to merge everything together.
The remarks::RemarkLinker class takes care of this and:
* Deduplicates remarks
* Filters remarks with no debug location
* Merges string tables from all the entries
As an output, it provides an iterator range that can be used to
serialize the remarks to a file.
Differential Revision: https://reviews.llvm.org/D69141
This helps with testing and debugging for paths that are assumed
absolute.
It also uses a FileError to provide the file path it's trying to open.
llvm-svn: 375008
It should be allowed to take a string table in case all the strings in
the remarks point there, but it shouldn't use it during serialization.
llvm-svn: 372042
The bitstream remark serializer landed in r367372.
This adds a bitstream remark parser that parser bitstream remark files
to llvm::remarks::Remark objects through the RemarkParser interface.
A few interesting things to point out:
* There are parsing helpers to parse the different types of blocks
* The main parsing helper allows us to parse remark metadata and open an
external file containing the encoded remarks
* This adds a dependency from the Remarks library to the BitstreamReader
library
* The testing strategy is to create a remark entry through YAML, parse
it, serialize it to bitstream, parse that back and compare the objects.
* There are close to no tests for malformed bitstream remarks, due to
the lack of textual format for the bitstream format.
* This adds a new C API for parsing bitstream remarks:
LLVMRemarkParserCreateBitstream.
* This bumps the REMARKS_API_VERSION to 1.
Differential Revision: https://reviews.llvm.org/D67134
llvm-svn: 371429
In order to keep remarks around, we need to make them tied to a string
table.
Users then can delete the parser and rely on the string table to keep
the memory of the strings alive and deduplicated.
llvm-svn: 371233
Add a new serializer, using a binary format based on the LLVM bitstream
format.
This format provides a way to serialize the remarks in two modes:
1) Separate mode: the metadata is separate from the remark entries.
2) Standalone mode: the metadata and the remark entries are in the same
file.
The format contains:
* a meta block: container version, container type, string table,
external file path, remark version
* a remark block: type, remark name, pass name, function name, debug
file, debug line, debug column, hotness, arguments (key, value, debug
file, debug line, debug column)
A string table is required for this format, which will be dumped in the
meta block to be consumed before parsing the remark blocks.
On clang itself, we noticed a size reduction of 13.4x compared to YAML,
and a compile-time reduction of between 1.7% and 3.5% on CTMark.
Differential Revision: https://reviews.llvm.org/D63466
Original llvm-svn: 367364
Revert llvm-svn: 367370
llvm-svn: 367372
Add a new serializer, using a binary format based on the LLVM bitstream
format.
This format provides a way to serialize the remarks in two modes:
1) Separate mode: the metadata is separate from the remark entries.
2) Standalone mode: the metadata and the remark entries are in the same
file.
The format contains:
* a meta block: container version, container type, string table,
external file path, remark version
* a remark block: type, remark name, pass name, function name, debug
file, debug line, debug column, hotness, arguments (key, value, debug
file, debug line, debug column)
A string table is required for this format, which will be dumped in the
meta block to be consumed before parsing the remark blocks.
On clang itself, we noticed a size reduction of 13.4x compared to YAML,
and a compile-time reduction of between 1.7% and 3.5% on CTMark.
Differential Revision: https://reviews.llvm.org/D63466
llvm-svn: 367364
The default mode is separate, where the metadata is serialized
separately from the remarks.
Another mode is the standalone mode, where the metadata is serialized
before the remarks, on the same stream.
llvm-svn: 367328
This adds support to the yaml remark parser to be able to parse remarks
directly from the metadata.
This supports parsing separate metadata and following the external file
with the associated metadata, and also a standalone file containing
metadata + remarks all together.
Original llvm-svn: 367148
Revert llvm-svn: 367151
This has a fix for gcc builds.
llvm-svn: 367155
This adds support to the yaml remark parser to be able to parse remarks
directly from the metadata.
This supports parsing separate metadata and following the external file
with the associated metadata, and also a standalone file containing
metadata + remarks all together.
llvm-svn: 367148
This allows every serializer format to implement metaSerializer() and
return the corresponding meta serializer.
Original llvm-svn: 366946
Reverted llvm-svn: 367004
This fixes the unit tests on Windows bots.
llvm-svn: 367078
Copying them is expensive. This allows the tables to be moved around at
lower cost, and allows a remarks::StringTable to be constructed from
a remarks::ParsedStringTable.
llvm-svn: 366864
This exposes better support to use a string table with a format through
an actual new remark::Format, called yaml-strtab.
This can now be used with -fsave-optimization-record=yaml-strtab.
llvm-svn: 366849
Before, everything was based on some kind of type erased parser
implementation which container a lot of boilerplate code when multiple
formats were to be supported.
This simplifies it by:
* the remark now owns its arguments
* *always* returning an error from the implementation side
* working around the way the YAML parser reports errors: catch them through
callbacks and re-insert them in a proper llvm::Error
* add a CParser wrapper that is used when implementing the C API to
avoid cluttering the C++ API with useless state
* LLVMRemarkParserGetNext now returns an object that needs to be
released to avoid leaking resources
* add a new API to dispose of a remark entry: LLVMRemarkEntryDispose
llvm-svn: 366217
* Add support for uniquing strings in the remark streamer and emitting the string table in the remarks section.
* Add parsing support for the string table in the RemarkParser.
From this remark:
```
--- !Missed
Pass: inline
Name: NoDefinition
DebugLoc: { File: 'test-suite/SingleSource/UnitTests/2002-04-17-PrintfChar.c',
Line: 7, Column: 3 }
Function: printArgsNoRet
Args:
- Callee: printf
- String: ' will not be inlined into '
- Caller: printArgsNoRet
DebugLoc: { File: 'test-suite/SingleSource/UnitTests/2002-04-17-PrintfChar.c',
Line: 6, Column: 0 }
- String: ' because its definition is unavailable'
...
```
to:
```
--- !Missed
Pass: 0
Name: 1
DebugLoc: { File: 3, Line: 7, Column: 3 }
Function: 2
Args:
- Callee: 4
- String: 5
- Caller: 2
DebugLoc: { File: 3, Line: 6, Column: 0 }
- String: 6
...
```
And the string table in the .remarks/__remarks section containing:
```
inline\0NoDefinition\0printArgsNoRet\0
test-suite/SingleSource/UnitTests/2002-04-17-PrintfChar.c\0printf\0
will not be inlined into \0 because its definition is unavailable\0
```
This is mostly supposed to be used for testing purposes, but it gives us
a 2x reduction in the remark size, and is an incremental change for the
updates to the remarks file format.
Differential Revision: https://reviews.llvm.org/D60227
llvm-svn: 359050
This adds a Remark class that allows us to share code when working with
remarks.
The C API has been updated to reflect this. Instead of the parser
generating C structs, it's now using a C++ object that is used through
opaque pointers in C. This gives us much more flexibility on what
changes we can make to the internal state of the object and interacts
much better with scenarios where the library is used through dlopen.
* C API updates:
* move from C structs to opaque pointers and functions
* the remark type is now an enum instead of a string
* unit tests updates:
* use mostly the C++ API
* keep one test for the C API
* rename to YAMLRemarksParsingTest
* a typo was fixed: AnalysisFPCompute -> AnalysisFPCommute.
* a new error message was added: "expected a remark tag."
* llvm-opt-report has been updated to use the C++ parser instead of the
C API
Differential Revision: https://reviews.llvm.org/D59049
Original llvm-svn: 356491
llvm-svn: 356519
This adds a Remark class that allows us to share code when working with
remarks.
The C API has been updated to reflect this. Instead of the parser
generating C structs, it's now using a C++ object that is used through
opaque pointers in C. This gives us much more flexibility on what
changes we can make to the internal state of the object and interacts
much better with scenarios where the library is used through dlopen.
* C API updates:
* move from C structs to opaque pointers and functions
* the remark type is now an enum instead of a string
* unit tests updates:
* use mostly the C++ API
* keep one test for the C API
* rename to YAMLRemarksParsingTest
* a typo was fixed: AnalysisFPCompute -> AnalysisFPCommute.
* a new error message was added: "expected a remark tag."
* llvm-opt-report has been updated to use the C++ parser instead of the
C API
Differential Revision: https://reviews.llvm.org/D59049
llvm-svn: 356491
Getting rid of the name "optimization remarks" for anything that
involves handling remarks on the client side.
It's safer to do this now, before we get stuck with that name in all the
APIs and public interfaces we decide to export to users in the future.
This renames llvm/tools/opt-remarks to llvm/tools/remarks-shlib, and now
generates `libRemarks.dylib` instead of `libOptRemarks.dylib`.
Differential Revision: https://reviews.llvm.org/D58535
llvm-svn: 355439
