This gives some aggregated information about the data access profiles.
The summaries are computed on the fly to save a profile version update.
Implementation-wise
* `MemProfSummary::printSummaryYaml` is updated to print data access
summaries for v4, the profile version that started to support data
access profiles.
* MemProfSummary.cpp has a FIXME comment to serialize the summary into
profile data, ideally batching with more substantial profile format
change.
* MemProfSummaryBuilder is not updated for now. This class is used to
serialize memprof summaries for v4 and above by memprof writer, and to
construct memprof summaries for v3 and prior versions by llvm-profdata.
Before llvm-profdata participated in llvm-driver it directly called
InitLLVM, which takes care of wildcard argument expansion for tools on
Windows. When llvm-driver support was added to llvm-profdata this
InitLLVM call was effectively moved into the common llvm-driver wrapper
mechanism.
More recently, in #162191, llvm-driver support was temporarily backed
out of llvm-profdata due to an issue with `cl::opt` handling. This
change reintroduces the direct call to InitLLVM in order to restore
wildcard expansion and also adds a test for the wildcard expansion on
Windows.
This change extends SampleFDO ext-binary and text format to record the
vtable symbols and their counts for virtual calls inside a function. The
vtable profiles will allow the compiler to annotate vtable types on IR
instructions and perform vtable-based indirect call promotion. An RFC is
in
https://discourse.llvm.org/t/rfc-vtable-type-profiling-for-samplefdo/87283
Given a function below, the before vs after of a function's profile is
illustrated in text format in the table:
```
__attribute__((noinline)) int loop_func(int i, int a, int b) {
Base *ptr = createType(i);
int sum = ptr->func(a, b);
delete ptr;
return sum;
}
```
| before | after |
| --- | --- |
| Samples collected in the function's body { <br> 0: 636241 <br> 1:
681458, calls: _Z10createTypei:681458 <br> 3: 543499, calls:
_ZN12_GLOBAL__N_18Derived24funcEii:410621 _ZN8Derived14funcEii:132878
<br> 5.1: 602201, calls: _ZN12_GLOBAL__N_18Derived2D0Ev:454635
_ZN8Derived1D0Ev:147566 <br> 7: 511057 <br> } | Samples collected in the
function's body { <br> 0: 636241 <br> 1: 681458, calls:
_Z10createTypei:681458 <br> 3: 543499, calls:
_ZN12_GLOBAL__N_18Derived24funcEii:410621 _ZN8Derived14funcEii:132878
<br> 3: vtables: _ZTV8Derived1:1377 _ZTVN12_GLOBAL__N_18Derived2E:4250
<br> 5.1: 602201, calls: _ZN12_GLOBAL__N_18Derived2D0Ev:454635
_ZN8Derived1D0Ev:147566 <br> 5.1: vtables: _ZTV8Derived1:227
_ZTVN12_GLOBAL__N_18Derived2E:765 <br> 7: 511057 <br> } |
Key points for this change:
1. In-memory representation of vtable profiles
* A field of type `map<LineLocation, map<FunctionId, uint64_t>>` is
introduced in a function's in-memory representation
[FunctionSamples](ccc416312e/llvm/include/llvm/ProfileData/SampleProf.h (L749-L754)).
2. The vtable counters for one LineLocation represents the relative
frequency among vtables for this LineLocation. They are not required to
be comparable across LineLocations.
3. For backward compatibility of ext-binary format, we take one bit from
ProfSummaryFlag as illustrated in the enum class `SecProfSummaryFlags`.
The ext-binary profile reader parses the integer type flag and reads
this bit. If it's set, the profile reader will parse vtable profiles.
4. The vtable profiles are optional in ext-binary format, and not
serialized out by default, we introduce an LLVM boolean option (named
`-extbinary-write-vtable-type-prof`). The ext-binary profile writer
reads the boolean option and decide whether to set the section flag bit
and serialize the in-memory class members corresponding to vtables.
5. This change doesn't implement `llvm-profdata overlap --sample` for
the vtable profiles. A subsequent change will do it to keep this one
focused on the profile format change.
We don't plan to add the vtable support to non-extensible format mainly
because of the maintenance cost to keep backward compatibility for prior
versions of profile data.
* Currently, the [non-extensible binary
format](5c28af4099/llvm/lib/ProfileData/SampleProfWriter.cpp (L899-L900))
does not have feature parity with extensible binary format today, for
instance, the former doesn't support [profile symbol
list](41e22aa31b/llvm/include/llvm/ProfileData/SampleProf.h (L1518-L1522))
or context-sensitive PGO, both of which give measurable performance
boost. Presumably the non-extensible format is not in wide use.
---------
Co-authored-by: Paschalis Mpeis <paschalis.mpeis@arm.com>
Hex escapes of the form \xff are not universally supported in the printf
implementations on the platforms that LLVM runs on (although they
apparently are in the shell builtins). Octal escapes are required to be
supported by POSIX. This patch converts all hex escapes to octal escapes
for compatibility reasons.
This came up when trying to turn on lit's internal shell by default for
llvm/. We started using /usr/bin/printf instead of the shell builtin on
MacOS, which does not support hex escapes.
I used the following python script to automate most of the conversion
with a few manual touchups needed:
```py
import sys
def process_line(to_process: str):
output = ""
i = 0
while i < len(to_process):
if to_process[i:i+2] == '\\x':
hex_string = to_process[i+2:i+4]
number = int(hex_string, 16)
output += "\\"
octal_string = oct(number)[2:]
if len(octal_string) == 1:
octal_string = "00" + octal_string
elif len(octal_string) == 2:
octal_string = "0" + octal_string
assert(len(octal_string) == 3)
output += octal_string
i += 4
else:
output += to_process[i]
i += 1
return output
with open(sys.argv[1]) as input_file:
lines = input_file.readlines()
for i, _ in enumerate(lines):
lines[i] = process_line(lines[i])
with open(sys.argv[1], 'w') as output_file:
output_file.writelines(lines)
```
This patch makes the AllocSites and CallSites sections optional in the
YAML format. This is useful for situations where a function has only
one section.
memprof::AllMemProfData has become home to multiple types of memprof
data -- heap profile and static data profile. When we write test
cases for static data profile in YAML, we do not want to include empty
heap profile. That would just add visual clutter.
This patch makes HeapProfileRecords optional.
`InstrProfWriter::addTemporalProfileTraces()` did not correctly account
for when the sources traces are sampled, but the reservoir size is
larger than what it was before, meaning there is room for more traces.
Also, if the reservoir size decreased, meaning traces should be
truncated.
Depends on https://github.com/llvm/llvm-project/pull/152550 for the test
refactor
Reapply #147854 after fixes merged in #151398.
Change memory access histogram storage from uint64_t to uint16_t to
reduce profile size on disk. This change updates the raw profile format
to v5. Also add a histogram test in compiler-rt since we didn't have one
before. With this change the histogram memprof raw for the basic test
reduces from 75KB -> 20KB.
Change memory access histogram storage from uint64_t to uint16_t to
reduce profile size on disk. This change updates the raw profile format
to v5. Also add a histogram test in compiler-rt since we didn't have one
before. With this change the histogram memprof raw for the basic test
reduces from 75KB -> 20KB.
The `--hot-func-list` flag is used for sample profiles to dump the list
of hot functions. Add support to dump hot functions for IRPGO profiles
as well.
This also removes a `priority_queue` used for `--topn`. We can instead
store all functions and sort at the end before dumping. Since we are
storing `StringRef`s, I believe this won't consume too much memory.
This patch adds support for a basic MemProf summary section, which is
built along with the indexed MemProf profile (e.g. when reading the raw
or YAML profiles), and serialized through the indexed profile just after
the header.
Currently only 6 fields are written, specifically the number of contexts
(total, cold, hot), and the max context size (cold, warm, hot).
To support forwards and backwards compatibility for added fields in the
indexed profile, the number of fields serialized first. The code is
written to support forwards compatibility (reading newer profiles with
additional summary fields), and comments indicate how to implement
backwards compatibility (reading older profiles with fewer summary
fields) as needed.
Support is added to print the summary as YAML comments when displaying
both the raw and indexed profiles via `llvm-profdata show`. Because they
are YAML comments, the YAML reader ignores these (the summary is always
recomputed when building the indexed profile as described above).
This necessitated moving some options and a couple of interfaces out of
Analysis/MemoryProfileInfo.cpp and into the new
ProfileData/MemProfSummary.cpp file, as we need to classify context
hotness earlier and also compute context ids to build the summary from
older indexed profiles.
Re-apply https://github.com/llvm/llvm-project/pull/139997 after fixing the use-of-uninitialized-memory error
(https://lab.llvm.org/buildbot/#/builders/94/builds/7373).
Tested: The error is reproduced with
https://github.com/llvm/llvm-zorg/blob/main/zorg/buildbot/builders/sanitizers/buildbot_bootstrap_msan.sh
without the fix, and test pass with the fix.
**Original commit message:**
https://github.com/llvm/llvm-project/pull/138170 introduces classes to
operate on data access profiles. This change supports the read and write
of `DataAccessProfData` in indexed format of MemProf (v4) as well as its
the text (yaml) format.
For indexed format:
* InstrProfWriter owns (by `std::unique_ptr<DataAccessProfData>`) the
data access profiles, and gives a non-owned copy when it calls
`writeMemProf`.
* MemProf v4 header has a new `uint64_t` to record the byte offset of
data access profiles. This `uint64_t` field is zero if data access
profile is not set (nullptr).
* MemProfReader reads the offset from v4 header and de-serializes
in-memory bytes into class `DataAccessProfData`.
For textual format:
* MemProfYAML.h adds the mapping for DAP class, and make DAP optional
for both read and write.
099a0fa (by @snehasish) introduces v4 which contains CalleeGuids in
CallSiteInfo, and this change changes the v4 format in place with data
access profiles. The current plan is to bump the version and enable v4
profiles with both features, assuming waiting for this change won't
delay the callsite change too long.
---------
Co-authored-by: Kazu Hirata <kazu@google.com>
https://github.com/llvm/llvm-project/pull/138170 introduces classes to
operate on data access profiles. This change supports the read and write
of `DataAccessProfData` in indexed format of MemProf (v4) as well as its
the text (yaml) format.
For indexed format:
* InstrProfWriter owns (by `std::unique_ptr<DataAccessProfData>`) the
data access profiles, and gives a non-owned copy when it calls
`writeMemProf`.
* MemProf v4 header has a new `uint64_t` to record the byte offset of
data access profiles. This `uint64_t` field is zero if data access
profile is not set (nullptr).
* MemProfReader reads the offset from v4 header and de-serializes
in-memory bytes into class `DataAccessProfData`.
For textual format:
* MemProfYAML.h adds the mapping for DAP class, and make DAP optional
for both read and write.
099a0fa (by @snehasish) introduces v4 which contains CalleeGuids in
CallSiteInfo, and this change changes the v4 format in place with data
access profiles. The current plan is to bump the version and enable v4
profiles with both features, assuming waiting for this change won't
delay the callsite change too long.
---------
Co-authored-by: Kazu Hirata <kazu@google.com>
This patch adds CalleeGuids to the serialized format and increments the version number to 4. The unit tests are updated to include a new test for v4 and the YAML format is also updated to be able to roundtrip the v4 format.
## Problem
When the build ids of the profile and binary do not match, the error
reported by llvm-profdata is `no entries in callstack map after
symbolization`, but the root cause of this problem is the **build id
mismatch**.
## Trigger scenario
For example, when performing `memprof` optimization on `clang`,
`rawprofile` is collected through `ninja clang`. In addition to running
clang, some other programs will also be executed, and these programs
will also generate rawprofile. When `no entries in callstack map after
symbolization` appears during `llvm-profdata merge`, users may
mistakenly think that the **instrumentation failed or other reasons**,
and will **not directly realize that the binary and profile do not
match**.
## Changed
Currently, when the build id does not match, an assert error is
triggered only in debug mode. Change it to directly return an error when
the build id does not match.
This pull request is the third part of an ongoing effort to extends PGO
instrumentation to GPU device code and depends on
https://github.com/llvm/llvm-project/pull/93365. This PR makes the
following changes:
- Allows PGO flags to be supplied to GPU targets
- Pulls version global from device
- Modifies `__llvm_write_custom_profile` and `lprofWriteDataImpl` to
allow the PGO version to be overridden
* Added YAML traits for `CallSiteInfo`
* Updated the `MemProfReader` to pass `Frames` instead of the entire
`CallSiteInfo`
* Updated test cases to use `testing::Field`
* Add YAML sequence traits for CallSiteInfo in MemProfYAML
* Also extend IndexedMemProfRecord
* XFAIL the MemProfYaml round trip test until we update the profile
format
For now we only read and write the additional information from the YAML
format. The YAML round trip test will be enabled when the serialized format is updated.
This patch does two things:
- During deserialization, we accept a function name for Frame as an
alternative to the usual GUID expressed as a hexadecimal number.
- During serialization, we print a GUID of Frame as a 16-digit
hexadecimal number prefixed with 0x in the usual way. (Without this
patch, we print a decimal number, which is not customary.)
The patch uses a machinery called "normalization" in YAML I/O, which
lets us serialize and deserialize into an alternative data structure.
For our use case, we have an alternative Frame data structure, which
is identical to "struct Frame" except that Function is of type
GUIDHex64 instead of GlobalValue::GUID. This alternative type
supports the two bullet points above without modifying "struct Frame"
at all.
This patch does two things:
- During deserialization, we accept a function name as an alternative
to the usual GUID represented as a hexadecimal number.
- During serialization, we print a GUID as a 16-digit hexadecimal
number prefixed with 0x in the usual way. (Without this patch, we
print a decimal number, which is not customary.)
In YAML, the MemProf profile is a vector of pairs of GUID and
MemProfRecord. This patch accepts a function name for the GUID, but
it does not accept a function name for the GUID used in Frames yet.
That will be addressed in a subsequent patch.
This patch adds YAML read/write support to llvm-profdata. The primary
intent is to accommodate MemProf profiles in test cases, thereby
avoiding the binary format.
The read support is via llvm-profdata merge. This is useful when we
want to verify that the compiler does the right thing on a given .ll
file and a MemProf profile in a test case. In the test case, we would
convert the MemProf profile in YAML to an indexed profile and invoke
the compiler on the .ll file along with the indexed profile.
The write support is via llvm-profdata show --memory. This is useful
when we wish to convert an indexed MemProf profile to YAML while
writing tests. We would compile a test case in C++, run it for an
indexed MemProf profile, and then convert it to the text format.
This patch removes MemProf format Version 0 now that version 2 and 3
seem to be working well.
I'm not touching version 1 for now because some tests still rely on
version 1.
Note that Version 0 is identical to Version 1 except that the MemProf
section of the indexed format has a MemProf version field.
The Lit in subject fails on z/OS since the input file
`Inputs/split-layout.profdata` is marked as ASCII even though it is
binary and the output file `Output/sample-split-layout.test.tmp-output`
is binary.
This PR removes the diff command which fails because it compares a
binary file and a text file. The rational is that this diff command
seems to be redundant to the `FileCheck` on the next command.
Using the flag `-split_layout` in llvm-profdata merge, the output
profile can write profiles with and without inlined function into two
different extbinary sections (and their FuncOffsetTable too). The
section without inlined functions are marked with `SecFlagFlat` and is
skipped by ThinLTO because it provides no useful info.
The split layout feature was already implemented in SampleProfWriter but
previously there is no way to use it from llvm-profdata.
PR 94264 had substantial increase in test binary size. This was due to
building LLVM and compiler-rt with GCC, implicitly causing the runtimes
to be built with full debuginfo. Rebuilding LLVM with
`-DCMAKE_CXX_COMPILER=/usr/bin/clang++` and regenerating .exe files with
script `update_memprof_inputs.sh` avoids this increase in test binary
size.
- The indexed iFDO profiles contains compressed vtable names for `llvm-profdata show --show-vtables` debugging
usage. An optimized build doesn't need it and doesn't decompress the blob now [1], since optimized binary has the
source code and IR to find vtable symbols.
- The motivation is to avoid increasing profile size when it's not necessary.
- This doesn't change the indexed profile format and thereby doesn't need a version change.
[1] eac925fb81/llvm/include/llvm/ProfileData/InstrProfReader.h (L696-L699)
Adds compile time flag -mllvm -memprof-histogram and runtime flag
histogram=true|false to turn Histogram collection on and off. The
-memprof-histogram flag relies on -memprof-use-callbacks=true to work.
Updates shadow mapping logic in histogram mode from having one 8 byte
counter for 64 bytes, to 1 byte for 8 bytes, capped at 255. Only
supports this granularity as of now.
Updates the RawMemprofReader and serializing MemoryInfoBlocks to binary
format, including changing to a new version of the raw binary format
from version 3 to version 4.
Updates creating MemoryInfoBlocks with and without Histograms. When two
MemoryInfoBlocks are merged, AccessCounts are summed up and the shorter
Histogram is removed.
Adds a memprof_histogram test case.
Initial commit for adding AccessCountHistograms up until RawProfile for
memprof
This patch adds Version 3 for development purposes. For now, this
patch adds V3 as a copy of V2.
For the most part, this patch adds "case Version3:" wherever "case
Version2:" appears. One exception is writeMemProfV3, which is copied
from writeMemProfV2 but updated to write out memprof::Version3 to the
MemProf header. We'll incrementally modify writeMemProfV3 in
subsequent patches.
The `llvm-profdata order` command is used to compute a function order
using traces from the input profile. Add the `--num-test-traces` flag to
keep aside N traces to evalute this order. These test traces are assumed
to be the actual function execution order in some experiment. The output
is a number that represents how many page faults we got. Lower is
better.
I tested on a large profile I already had.
```
llvm-profdata order default.profdata --num-test-traces=30
# Ordered 149103 functions
# Total area under the page fault curve: 2.271827e+09
...
```
I also improved `TemporalProfTraceTy::createBPFunctionNodes()` in a few
ways:
* Simplified how `UN`s are computed
* Change how the initial `Node` order is computed
* Filter out rare and common `UN`s
* Output vector is an aliased argument instead of a return
These changes slightly improved the evaluation in my test.
```
llvm-profdata order default.profdata --num-test-traces=30
# Ordered 149103 functions
# Total area under the page fault curve: 2.268586e+09
...
```
The `--temporal-profile-max-trace-length=0` flag in the `llvm-profdata
merge` command is used to remove traces from a profile. There was a bug
where traces would not be cleared if the profile was already sampled.
This patch fixes that.
Curently, the compiler only uses several fields of MemoryInfoBlock.
Serializing all fields into the indexed MemProf file simply wastes
storage.
This patch limits the schema down to four fields for Version2 by
default. It retains the old behavior of serializing all fields via:
llvm-profdata merge --memprof-version=2 --memprof-full-schema
This patch reduces the size of the indexed MemProf profile I have by
40% (1.6GB down to 1.0GB).
This patch adds Version2 of the indexed MemProf format. The new
format comes with a hash table from CallStackId to actual call stacks
llvm::SmallVector<FrameId>. The rest of the format refers to call
stacks with CallStackId. This "values + references" model effectively
deduplicates call stacks. Without this patch, a large indexed memprof
file of mine shrinks from 4.4GB to 1.6GB, a 64% reduction.
This patch does not make Version2 generally available yet as I am
planning to make a few more changes to the format.
(The profile format change is split into a standalone change into https://github.com/llvm/llvm-project/pull/81691)
* For InstrFDO value profiling, implement instrumentation and lowering for virtual table address.
* This is controlled by `-enable-vtable-value-profiling` and off by default.
* When the option is on, raw profiles will carry serialized `VTableProfData` structs and compressed vtables as payloads.
* Implement profile reader and writer support
* Raw profile reader is used by `llvm-profdata` but not compiler. Raw profile reader will construct InstrProfSymtab with symbol names, and map profiled runtime address to vtable symbols.
* Indexed profile reader is used by `llvm-profdata` and compiler. When initialized, the reader stores a pointer to the beginning of in-memory compressed vtable names and the length of string. When used in `llvm-profdata`, reader decompress the string to show symbols of a profiled site. When used in compiler, string decompression doesn't
happen since IR is used to construct InstrProfSymtab.
* Indexed profile writer collects the list of vtable names, and stores that to index profiles.
* Text profile reader and writer support are added but mostly follow the implementation for indirect-call value type.
* `llvm-profdata show -show-vtables <args> <profile>` is implemented.
rfc in
https://discourse.llvm.org/t/rfc-dynamic-type-profiling-and-optimizations-in-llvm/74600#pick-instrumentation-points-and-instrument-runtime-types-7
