We already have code to extract bitcode files from archives so they can
be distributed. Extend this code to extract bitcode from FatLTO objects
too, which otherwise cannot be used with DTLTO.
DTLTO emits temporary files to allow distribution of archive member
inputs.
It also emits temporary files from the ThinLTO backend, such as the
index files needed for each distributed ThinLTO backend compilation.
This change brings archive member temporary files into line with those
produced by the ThinLTO backend. They are now emitted in the same
location, warnings are emitted if they cannot be deleted, and they are
cleaned up on abnormal exit (e.g. Ctrl-C). All temporary files are
preserved when --save-temps is specified.
The existing signal-handling test has been extended to cover the full
set of DTLTO temporary files, and a new test has been added to exercise
temporary file handling in normal operation. Additionally, a minimal
test has been added to show the COFF behaviour.
SIE Internal tracker: TOOLCHAIN-21022
Add time-trace scopes to the DTLTO-specific input-handling code to
improve observability and debugging.
These scopes are tested via LLD, as the primary purpose of this code is
to support member files of non-thin archives as DTLTO inputs.
`llvm-lto2` does not currently support archives. Adding archive support
to `llvm-lto2` solely for testing these scopes does not appear to be
worthwhile.
As part of this change, the deletion of temporary DTLTO input files has
been moved. Cleanup now occurs after LTO has completed, rather than
during destruction of the LTO object. This is required since by the time
the LTO object is destroyed, time-traces have already been finalized, so
no additional trace data can be recorded.
Recording time-trace data for temporary file deletion is important, as
this has been a source of performance issues in the past and an area
where we expect to make further performance improvements if supported by
the data.
SIE internal tracker: TOOLCHAIN-21021
This change fixes two issues when processing multi-module bitcode files
in DTLTO:
1. The DTLTO archive handling code incorrectly uses
getSingleBitcodeModule(), which asserts when the bitcode file contains
more than one module.
2. The temporary file containing the contents of an input archive member
was not emitted for multi-module bitcode files. This was due to
incorrect logic for recording whether a bitcode input contains any
ThinLTO modules. In a typical multi-module bitcode file, the first
module is a ThinLTO module while a subsequent auxiliary module is
non-ThinLTO. When modules are processed in order, the auxiliary module
causes the entire bitcode file to be classified as non-ThinLTO, and the
archive-member emission logic then incorrectly skips it.
In addition, this patch adds a test that verifies that multi-module
bitcode files can be successfully linked with DTLTO. The test reproduces
both issues as they existed prior to this change.
SIE Tracker: TOOLCHAIN-21008
This patch implements support for handling archive members in DTLTO.
Unlike ThinLTO, where archive members are passed as in-memory buffers,
DTLTO requires archive members to be materialized as individual files on
the filesystem.
This is necessary because DTLTO invokes clang externally, which expects
file-based inputs.
To support this, this implementation identifies archive members among
the input files,
saves them to the filesystem, and updates their module_id to match their
file paths.
