Remove the paper trail warning from dsymutil and the DWARF linker. The
original purpose of this functionality was to leave a paper trail in the
output artifact about missing object files. The current implementation
however has diverged and is the source of a pretty serious bug:
- In the debug map parser, missing object files are not the only
warnings we emit. When paper trail warnings are enabled, all of them end
up in the dSYM which wasn't the goal.
- When warnings are associated with a object file in the debug map, it
is skipped by the DWARF linker. This only makes sense if the object file
is missing and is obviously incorrect for any other type of warning
(such as a missing symbol).
The combination of the two means that we can generate broken DWARF when
the feature is enabled. AFAIK it was only used by Apple and nobody I
spoke to has relied on it, so rather than fixing the broken behavior I
propose we remove it.
This parallels the binutils/BSD flag of the same name. Debugging
information is loaded to print line number information for symbols.
Defined symbols are symbolized by their section addresses, and undefined
symbols by their first text reloc with line info.
Differential Revision: https://reviews.llvm.org/D150987
This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.
Motivation
----------
D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv. For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:
```
Script:
--
: 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
--
Exit Code: 2
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello"
# command output:
hello
$ "FileCheck" "bogus.txt"
# command stderr:
Could not open check file 'bogus.txt': No such file or directory
error: command failed with exit status: 2
--
```
In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window. The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:
- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
are not clearly delineated.
- They don't always use proper shell quoting. Instead, they blindly
enclose all command-line arguments in double quotes.
Changes
-------
D154984 plus this patch converts the above example into:
```
Exit Code: 2
Command Output (stdout):
--
# RUN: at line 1
echo hello | FileCheck bogus-file.txt && echo success
# executed command: echo hello
# .---command stdout------------
# | hello
# `-----------------------------
# executed command: FileCheck bogus-file.txt
# .---command stderr------------
# | Could not open check file 'bogus-file.txt': No such file or directory
# `-----------------------------
# error: command failed with exit status: 2
--
```
Thus, this patch addresses the above issues as follows:
- The entire execution trace can be copied and pasted in bulk to a
terminal for correct execution of the RUN lines, which are printed
intact as they appeared in the original RUN lines except lit
substitutions are expanded. Everything else in the execution trace
appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
it executes to show (1) that the command actually executed (e.g.,
`echo success` above didn't) and (2) what stdout, stderr, non-zero
exit status, and output files are associated with the command, if
any. Shell quoting in the command is now correct and minimal but is
not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
file is now delineated clearly in the trace.
To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command. It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly. For now, `@echo` isn't documented for use
in lit tests.
Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.
(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)
Caveat
------
This patch only makes the above changes for lit's internal shell. In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.
To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:
```
config.test_format = lit.formats.ShTest(execute_external=False)
```
Reviewed By: MaskRay, awarzynski
Differential Revision: https://reviews.llvm.org/D156954
This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.
**Motivation**: -a shows output from all tests, and -v shows output
from just failed tests. Without this patch, that output from each
test includes a section called "Script:", which includes all shell
commands that lit has computed from RUN directives and will attempt to
run for that test. The effect of -vv (which also implies -v if
neither -a or -v is specified) is to extend that output with shell
commands as they are executing so you can easily see which one failed.
For example, when using lit's internal shell and -vv:
```
Script:
--
: 'RUN: at line 1'; echo hello world
: 'RUN: at line 2'; 3c40 hello world
: 'RUN: at line 3'; echo hello world
--
Exit Code: 127
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello" "world"
hello world
$ ":" "RUN: at line 2"
$ "3c40" "hello" "world"
'3c40': command not found
error: command failed with exit status: 127
--
```
Notice that all shell commands that actually execute appear in the
output twice, once for "Script:" and once for -vv. Especially for
tests with many RUN directives, the result is noisy. When searching
through the output for a particular shell command, it is easy to get
lost and mistake shell commands under "Script:" for shell commands
that actually executed.
**Change**: With this patch, a test's output changes in two ways.
First, the "Script:" section is never shown. Second, omitting -vv no
longer disables printing of shell commands as they execute. That is,
-a and -v imply -vv, and so -vv is deprecated as it is just an alias
for -v.
**Secondary motivation**: We are also working to introduce a PYTHON
directive, which can appear between RUN directives. How should PYTHON
directives be represented in the "Script:" section, which has
previously been just a shell script? We could probably think of
something, but adding info about PYTHON directive execution in the -vv
trace seems more straight-forward and more useful.
(This patch also removes a confusing point in the -vv documentation:
at least when using bash as an external shell, -vv echoes commands to
the shell's stderr not stdout.)
Reviewed By: awarzynski, Endill, ldionne, MaskRay
Differential Revision: https://reviews.llvm.org/D154984
GNU readelf introduced --extra-sym-info/-X to display the section name
for --syms (https://sourceware.org/PR30684). Port the feature, which is
currently llvm-readelf only.
For STO_AARCH64_VARIANT_PCS/STO_RISCV_VARIANT_PCS, the Ndx and Name
columns may not be aligned.
Enable color highlighting of disassembly in llvm-objdump. This patch
introduces a new flag --disassembler-color=<mode> that enables or
disables highlighting disassembly with ANSI escape codes. The default
mode is to enable color highlighting if outputting to a color-enabled
terminal.
Differential revision: https://reviews.llvm.org/D159224
Add support for syntax highlighting assembly. The patch introduces new
RAII helper called WithMarkup that takes care of both emitting colors
and markup annotations. It makes adding markup easier and ensures colors
and annotations remain consistent.
This patch adopts the new helper in the AArch64 backend. If your backend
already uses markup annotations, adoption is as easy as using the new
MCInstPrinter::markup overload.
Differential revision: https://reviews.llvm.org/D159162
The MatchTable-based GlobalISel Combiner backend is the new default. There are no in-tree users left of the old backend.
- Removed implementation of old MatchDAG-based Combiner, including tests, the backend itself and all supporting code.
- Renamed MatchTable backend to `GlobalISelCombinerEmitter.cpp` + removed "-matchtable" from its CL option.
- no need to have a verbose name as it's the only backend left now.
Reviewed By: aemerson
Differential Revision: https://reviews.llvm.org/D158710
If llvm-symbolizer finds a malformed command, it echoes it to the
standard output. New versions of binutils (starting from 2.39) allow to
specify an address by a symbols. Implementation of this feature in
llvm-symbolizer makes the current reaction on invalid input
inappropriate. Almost any invalid command may be treated as a symbol
name, so the right reaction should be "symbol not found" in such case.
The exception are commands that are recognized but have incorrect
syntax, like "FILE:FILE:". The utility must produce descriptive
diagnostic for such input and route it to the stderr.
This change implements the new reaction on invalid input and is a
prerequisite for implementation of symbol lookup in llvm-symbolizer.
Differential Revision: https://reviews.llvm.org/D157210
This patch and D156954 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.
**Motivation**: -a shows output from all tests, and -v shows output
from just failed tests. Without this patch, that output from each
test includes a section called "Script:", which includes all shell
commands that lit has computed from RUN directives and will attempt to
run for that test. The effect of -vv (which also implies -v if
neither -a or -v is specified) is to extend that output with shell
commands as they are executing so you can easily see which one failed.
For example, when using lit's internal shell and -vv:
```
Script:
--
: 'RUN: at line 1'; echo hello world
: 'RUN: at line 2'; 3c40 hello world
: 'RUN: at line 3'; echo hello world
--
Exit Code: 127
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "echo" "hello" "world"
hello world
$ ":" "RUN: at line 2"
$ "3c40" "hello" "world"
'3c40': command not found
error: command failed with exit status: 127
--
```
Notice that all shell commands that actually execute appear in the
output twice, once for "Script:" and once for -vv. Especially for
tests with many RUN directives, the result is noisy. When searching
through the output for a particular shell command, it is easy to get
lost and mistake shell commands under "Script:" for shell commands
that actually executed.
**Change**: With this patch, a test's output changes in two ways.
First, the "Script:" section is never shown. Second, omitting -vv no
longer disables printing of shell commands as they execute. That is,
-a and -v imply -vv, and so -vv is deprecated as it is just an alias
for -v.
**Secondary motivation**: We are also working to introduce a PYTHON
directive, which can appear between RUN directives. How should PYTHON
directives be represented in the "Script:" section, which has
previously been just a shell script? We could probably think of
something, but adding info about PYTHON directive execution in the -vv
trace seems more straight-forward and more useful.
(This patch also removes a confusing point in the -vv documentation:
at least when using bash as an external shell, -vv echoes commands to
the shell's stderr not stdout.)
Reviewed By: awarzynski, Endill, ldionne, MaskRay
Differential Revision: https://reviews.llvm.org/D154984
This is a GSoC 2023 project ([discourse link](https://discourse.llvm.org/t/coverage-support-a-hierarchical-directory-structure-in-generated-coverage-html-reports/68239)).
llvm-cov currently generates a single top-level index HTML file, which causes rendering scalability issues in large projects. This patch adds support for hierarchical directory structure into the HTML reports to solve scalability issues by introducing the following changes:
- Added a new command line option `--show-directory-coverage` for `llvm-cov show`. It works both for `--format=html` and `--format=text`.
- Two new classes: `CoveragePrinterHTMLDirectory` and `CoveragePrinterTextDirectory` was added to support the new option.
- A tool class `DirectoryCoverageReport` was added to support the two classes above.
- Updated the document.
- Added a new regression test for `--show-directory-coverage`.
Reviewed By: phosek, gulfem
Differential Revision: https://reviews.llvm.org/D151283
Previously the --path-equivalence parameter would allow to specify a single remap pair (coverage data path - local source file path). This patch changes this allowing to pass as many remaps as needed.
Reviewed By: keith
Differential Revision: https://reviews.llvm.org/D154223
`llvm-cov convert-for-testing` only functions properly when the input binary contains a single source file. When the binary has multiple source files, a `Malformed coverage data` error will occur when the generated .covmapping is read back. This is because the testing format lacks support for indicating the size of its file records, and current implementation just assumes there's only one record in it. This patch fixes this problem by introducing a new testing format version.
Changes to the code:
- Add a new format version. The version number is stored in the the last 8 bytes of the orignial magic number field to be backward-compatible.
- Output a LEB128 number before the file records section to indicate its size in the new version.
- Change the format parsing code correspondingly.
- Update the document to formalize the testing format.
- Additionally, fix the bug when converting COFF binaries.
Reviewed By: phosek, gulfem
Differential Revision: https://reviews.llvm.org/D156611
This reverts commit a5fe6c7f5e2d1d265bd7c312ef55259fee7a68f9.
This change is causing problems with Windows build bots due to a hanging zombie llvm-symbolizer.exe process.
If llvm-symbolizer finds a malformed command, it echoes it to the
standard output. New versions of binutils (starting from 2.39) allow to
specify an address by a symbols. Implementation of this feature in
llvm-symbolizer makes the current reaction on invalid input
inappropriate. Almost any invalid command may be treated as a symbol
name, so the right reaction should be "symbol not found" in such case.
The exception are commands that are recognized but have incorrect
syntax, like "FILE:FILE:". The utility must produce descriptive
diagnostic for such input and route it to the stderr.
This change implements the new reaction on invalid input and is a
prerequisite for implementation of symbol lookup in llvm-symbolizer.
Differential Revision: https://reviews.llvm.org/D157210
Currently the llvm-exegesis docs use a mix of double dash and single
dash options with seemingly no pattern. This patch makes everything
double dash options as it has been suggested that we should be
advertising double dash long options exclusively in the documentation.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D157641
Currently the documentation on the command line options for llvm-link is
quite sparse. This patch adds in the options that the tool understands
that aren't currently present in the documentation.
Reviewed By: tejohnson, MaskRay
Differential Revision: https://reviews.llvm.org/D155904
Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.
Changes lit logic to:
Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.
Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.
Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).
How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows
Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir
Patch by Tristan Labelle <tristan@thebrowser.company>!
This patch is the first part of https://llvm.org/OpenProjects.html#llvm_patch_coverage.
We have first define a new variable LLVM_TEST_COVERAGE which when set, pass --per-test-coverage option to
llvm-lit which will help in setting a unique value to LLVM_PROFILE_FILE for each RUN. So for example
coverage data for test case llvm/test/Analysis/AliasSet/memtransfer.ll will be emitted as
build/test/Analysis/AliasSet/memtransfer0.profraw
Reviewed By: hnrklssn
Differential Revision: https://reviews.llvm.org/D154280
Currently, objcopy cannot set the new flag SHF_X86_64_LARGE. This change introduces the named flag "large" which translates to that section flag.
An "invalid argument" error is produced if a user attempts to set the flag on an architecture other than X86_64.
Reviewed By: jhenderson, MaskRay
Differential Revision: https://reviews.llvm.org/D153262
This patch is the first part of https://llvm.org/OpenProjects.html#llvm_patch_coverage.
We have first define a new variable LLVM_TEST_COVERAGE which when set, pass --emit-coverage option to
llvm-lit which will help in setting a unique value to LLVM_PROFILE_FILE for each RUN. So for example
coverage data for test case llvm/test/Analysis/AliasSet/memtransfer.ll will be emitted as
build/test/Analysis/AliasSet/memtransfer.profraw
Reviewed By: hnrklssn
Differential Revision: https://reviews.llvm.org/D154280
This option had originally been added in D83069 to allow disabling the
check that something is going to get run at all when a specific test name
is used on the command-line. Since we now use getTestsForPath() (from D151664)
to get the tests to run for a specific path, we don't need a specific check
for this anymore -- Lit will produce the same complaint it would produce if
you provided a directory with no tests.
If one needs to run a specific test on the command-line and the Lit
configuration would normally not include that test, the configuration
should be set up as a "standalone" configuration or it should be fixed
to allow for that test to be found (i.e. probably fix the allowed test
suffixes).
Differential Revision: https://reviews.llvm.org/D153967
Extend D127824 to the 32-bit Power architecture.
AFAICT GNU objdump -d dumps all instructions for 32-bit as well.
Reviewed By: #powerpc, nemanjai
Differential Revision: https://reviews.llvm.org/D155114
Summary:
Adding a new option -traceback-table to print out the traceback info of xcoff ojbect file.
Reviewers: James Henderson, Fangrui Song, Stephen Peckham, Xing Xue
Differential Revision: https://reviews.llvm.org/D89049
Switch the parse of command line options fromllvm::cl to OptTable.
The motivation for this change is to continue adding llvm based tools
to the llvm driver multicall.
Differential Revision: https://reviews.llvm.org/D153665
In [0] we described an algorithm called //BalancedPartitioning// (bp) to consume function traces [1] and compute a function order that reduces the number of page faults during startup.
This patch adds the `order` command to the `llvm-profdata` tool which uses bp to output a function order that can be passed to the linker via `--symbol-ordering-file=`.
Special thanks to Sergey Pupyrev and Julian Mestre for designing this balanced partitioning algorithm.
[0] https://discourse.llvm.org/t/rfc-temporal-profiling-extension-for-irpgo/68068
[1] https://reviews.llvm.org/D147287
Reviewed By: spupyrev
Differential Revision: https://reviews.llvm.org/D147812
The comment moved is referring to the --output-asm-syntax flag rather
than the --print-imm-hex flag, but seems to have mistakenly been put
under the definition of that flag due to some misplaced line numbers on
phabricator.
At the moment, dsymutil drops all remarks without debug location.
There are many cases where debug location may be missing for remarks,
mostly due LLVM not preserving debug locations. When using bitstream
remarks for statistical analysis, those missed remarks mean we get an
incomplete picture.
The patch flips the default to keeping all remarks and leaving it to
tools that display remarks to filter out remarks without debug locations
as needed.
The new --remarks-drop-without-debug flag can be used to drop remarks
without debug locations, i.e. restore the previous behavior.
Reviewed By: thegameg
Differential Revision: https://reviews.llvm.org/D151089
llvm-exegesis has both a capture mode and an analysis mode that can be
used independently of each other. This patch makes it clear that
analysis mode will work on other platforms that LLVM supports in the
documentation which was unclear before.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D150536
Currently, there is no documentation on what platforms and architectures
llvm-exegesis is supported on. This patch adds in user-facing
documentation in the CommandGuide about what architectures are supported
as well as developer facing documentation detailing the technical
reasons for why certain platforms are supported and some aren't.
This is a follow-up after discussion in
https://discourse.llvm.org/t/clarification-on-platform-support-for-llvm-exegesis/70206.
Reviewed By: kpdev42
Differential Revision: https://reviews.llvm.org/D149378
Since we don't always need the vendor extension to be in riscv_vector.td,
so it's better to make it be in separated header.
Depends on D148223 and D148680
Differential Revision: https://reviews.llvm.org/D148308
As discussed in [0], add a `weight` field to temporal profiling traces found in profiles. This allows users to use the `--weighted-input=` flag in the `llvm-profdata merge` command to weight traces from different scenarios differently.
Note that this is a breaking change, but since [1] landed very recently and there is no way to "use" this trace data, there should be no users of this feature. We believe it is acceptable to land this change without bumping the profile format version.
[0] https://reviews.llvm.org/D147812#4259507
[1] https://reviews.llvm.org/D147287
Reviewed By: snehasish
Differential Revision: https://reviews.llvm.org/D148150
This patch adds in documentation for the --use-dummy-perf-counters
option (introduced in D146301).
Reviewed By: kpdev42
Differential Revision: https://reviews.llvm.org/D147842
D147710 introduced a new annotation-count subcommand to llvm-remarkutil
and added in documentation. However, the reference added under the
subcommands list never actually pointed to anything. This patch adds a
marker for the reference to point to so that the link works and the
sphinx build finishes without any errors.
