139 Commits

Author SHA1 Message Date
Peter Collingbourne
3afc2be1f0
llvm-lto2: Add print-guid subcommand.
This is useful for debugging ThinLTO issues.

Reviewers: teresajohnson

Reviewed By: teresajohnson

Pull Request: https://github.com/llvm/llvm-project/pull/143992
2025-06-13 14:35:40 -07:00
bd1976bris
6520b21ce0
[DTLTO][LLVM] Integrated Distributed ThinLTO (DTLTO) (#127749)
This patch adds initial support for Integrated Distributed ThinLTO
(DTLTO) in LLVM, which manages distribution internally during the
traditional link step. This enables compatibility with any build
system that supports in-process ThinLTO. In contrast, existing
approaches to distributed ThinLTO, which split the thin-link
(--thinlto-index-only), backend compilation, and final link into
separate steps, require build system support, e.g. Bazel.

This patch implements the core DTLTO mechanism, which enables
delegation of ThinLTO backend jobs to an external process (the
distributor). The distributor can then manage job distribution through
systems like Incredibuild. A generic JSON interface is used to
communicate with the distributor, allowing for the creation of new
distributors (and thus integration with different distribution
systems) without modifying LLVM.

Please see llvm/docs/dtlto.rst for more details.

RFC: https://discourse.llvm.org/t/rfc-integrated-distributed-thinlto/69641
Design Review: https://github.com/llvm/llvm-project/pull/126654
2025-05-23 20:07:53 +01:00
anjenner
c3f815ba82
Modify the localCache API to require an explicit commit on CachedFile… (#136121)
…Stream.

CachedFileStream has previously performed the commit step in its
destructor, but this means its only recourse for error handling is
report_fatal_error. Modify this to add an explicit commit() method, and
call this in the appropriate places with appropriate error handling for
the location.

Currently the destructor of CacheStream gives an assert failure in Debug
builds if commit() was not called. This will help track down any
remaining uses of the API that assume the old destructior behaviour. In
Release builds we fall back to the previous behaviour and call
report_fatal_error if the commit fails.

This is version 2 of this PR, superseding reverted PR
https://github.com/llvm/llvm-project/pull/115331 . I have incorporated a
change to the testcase to make it more reliable on Windows, as well as
two follow-up changes
(df79000896
and
b0baa1d8bd)
that were also reverted when 115331 was reverted.

---------

Co-authored-by: Augie Fackler <augie@google.com>
Co-authored-by: Vitaly Buka <vitalybuka@google.com>
2025-04-22 09:45:15 +01:00
Jeremy Morse
40f9bb9e25
[DebugInfo][RemoveDIs] Eliminate another debug-info variation flag (#133917)
The "preserve input debug-info format" flag allowed some tooling to opt
into not seeing the new debug records yet, and to not autoupgrade. This
was good at the time, but un-necessary now that we'll be ditching
intrinsics shortly.

It also hides errors now: verify-uselistorder was hardcoding this flag
to on, and as a result it hasn't seen debug records before. Thus, we
missed a uselistorder variation: constant-expressions such as GEPs can
be contained within debug records and completely isolated from the value
hierachy, see the metadata-use-uselistorder.ll test. These Values didn't
get ordered, but were legitimate uses of constants like "i64 0", and we
now run into difficulty handling that. The patch to AsmWriter seeks
Values to order even through debug-info now.

Finally there are a few intrinsics-tests relying on this flag that we
can just delete, such as one in llvm-reduce and another few in the
LocalTest unit tests. For the fast-isel test, it was added in
https://reviews.llvm.org/D67703 explicitly for checking the size of
blocks without debug-info and in 1525abb9c94 the codepath it tests moved
towards being sunsetted. It'll be totally redundant once RemoveDIs is on
permanently.

Note that there's now no explicit test for the textual-IR autoupgrade
path. I submit that we can rely on the thousands of .ll files where
we've only been bothered to update the outputs, not the inputs, to debug
records.
2025-04-09 18:00:28 +01:00
Jeremy Morse
1ebc308bba
[DebugInfo][RemoveDIs] Remove debug-intrinsic printing cmdline options (#131855)
During the transition from debug intrinsics to debug records, we used
several different command line options to customise handling: the
printing of debug records to bitcode and textual could be independent of
how the debug-info was represented inside a module, whether the
autoupgrader ran could be customised. This was all valuable during
development, but now that totally removing debug intrinsics is coming
up, this patch removes those options in favour of a single flag
(experimental-debuginfo-iterators), which enables autoupgrade, in-memory
debug records, and debug record printing to bitcode and textual IR.

We need to do this ahead of removing the
experimental-debuginfo-iterators flag, to reduce the amount of
test-juggling that happens at that time.

There are quite a number of weird test behaviours related to this --
some of which I simply delete in this commit. Things like
print-non-instruction-debug-info.ll , the test suite now checks for
debug records in all tests, and we don't want to check we can print as
intrinsics. Or the update_test_checks tests -- these are duplicated with
write-experimental-debuginfo=false to ensure file writing for intrinsics
is correct, but that's something we're imminently going to delete.

A short survey of curious test changes:
* free-intrinsics.ll: we don't need to test that debug-info is a zero
cost intrinsic, because we won't be using intrinsics in the future.
* undef-dbg-val.ll: apparently we pinned this to non-RemoveDIs in-memory
mode while we sorted something out; it works now either way.
* salvage-cast-debug-info.ll: was testing intrinsics-in-memory get
salvaged, isn't necessary now
* localize-constexpr-debuginfo.ll: was producing "dead metadata"
intrinsics for optimised-out variable values, dbg-records takes the
(correct) representation of poison/undef as an operand. Looks like we
didn't update this in the past to avoid spurious test differences.
* Transforms/Scalarizer/dbginfo.ll: this test was explicitly testing
that debug-info affected codegen, and we deferred updating the tests
until now. This is just one of those silent gnochange issues that get
fixed by RemoveDIs.

Finally: I've added a bitcode test, dbg-intrinsics-autoupgrade.ll.bc,
that checks we can autoupgrade debug intrinsics that are in bitcode into
the new debug records.
2025-04-01 14:27:11 +01:00
Kazu Hirata
8db434a481
[bugpoint] Avoid repeated hash lookups (NFC) (#133616) 2025-03-29 22:40:05 -07:00
Mingming Liu
d99033e4b4
[LTO][WPD] Suppress WPD on a class if the LTO unit doesn't have the prevailing definition of this class (#131721)
Before this patch, whole program devirtualization is suppressed on a
class if any superclass is visible to regular object files, by recording
the class GUID in `VisibleToRegularObjSymbols`.

This patch suppresses whole program devirtualization on a class if the
LTO unit doesn't have the prevailing definition of this class (e.g., the
prevailing definition is in a shared library)

Implementation summaries:
1. In llvm/lib/LTO/LTO.cpp, `IsVisibleToRegularObj` is updated to look
at the global resolution's `IsPrevailing` bit for ThinLTO and
regularLTO.
2. In llvm/tools/llvm-lto2/llvm-lto2.cpp, 
- three command line options are added so `llvm-lto2` can override
`Conf.HasWholeProgramVisibility`, `Conf.ValidateAllVtablesHaveTypeInfos`
and `Conf.AllVtablesHaveTypeInfos`.
    
The test case is reduced from a small C++ program (main.cc, lib.cc/h
pasted below in [1]). To reproduce the program failure without this
patch, compile lib.cc into a shared library, and provide it to a ThinLTO
build of main.cc (commands are pasted in [2]).

[1]

* lib.h

```
#include <cstdio>

class Derived {
 public:
  void dispatch();
  virtual void print();
  virtual void sum();
};

void Derived::dispatch() {
  static_cast<Derived*>(this)->print();
  static_cast<Derived*>(this)->sum();
}

void Derived::sum() {
  printf("Derived::sum\n");
}

__attribute__((noinline)) void* create(int i);
__attribute__((noinline)) void* getPtr(int i);

```

* lib.cc

```
#include "lib.h"
#include <cstdio>
#include <iostream>

class Derived2 : public Derived {
public:
  void print() override {
    printf("DerivedSharedLib\n");
  }
  void sum() override {
    printf("DerivedSharedLib::sum\n");
  }
};


void Derived::print() {
  printf("Derived\n");
}

__attribute__((noinline)) void* create(int i) {
  if (i & 1)
    return new Derived2();
  return new Derived();
}

```

* main.cc

```
cat main.cc
#include "lib.h"

class DerivedN : public Derived {
 public:
};

__attribute__((noinline)) void* getPtr(int x) {
  return new DerivedN();
}


int main() {
  Derived*b = static_cast<Derived*>(create(201));
  b->dispatch();
  delete b;

  Derived* a = static_cast<Derived*>(getPtr(202));
  a->dispatch();
  delete a;
  return 0;
}

```

[2]
```
# compile lib.o in a shared library.
$ ./bin/clang++ -O2 -fPIC -c lib.cc -o lib.o
$ ./bin/clang++ -shared -o libdata.so lib.o

# Provide the shared library in `-ldata`
$ ./bin/clang++ -v -g -ldata --save-temps -fno-discard-value-names -Wl,-mllvm,-print-before=wholeprogramdevirt -Wl,-mllvm,-wholeprogramdevirt-check=trap -Rpass=wholeprogramdevirt -Wl,--lto-whole-program-visibility -Wl,--lto-validate-all-vtables-have-type-infos -mllvm -disable-icp=true -Wl,-mllvm,-disable-icp=false -flto=thin -fwhole-program-vtables  -fno-split-lto-unit -fuse-ld=lld   main.cc  -L . -o main >/tmp/wholeprogramdevirt.ir 2>&1

# Run the program hits a segmentation fault with  `-Wl,-mllvm,-wholeprogramdevirt-check=trap`

$ LD_LIBRARY_PATH=. ./main
DerivedSharedLib
Trace/breakpoint trap (core dumped)

```
2025-03-19 22:10:57 -07:00
Jeremy Morse
792a6f8119
[RemoveDIs] Remove "try-debuginfo-iterators..." test flags (#130298)
These date back to when the non-intrinsic format of variable locations
was still being tested and was behind a compile-time flag, so not all
builds / bots would correctly run them. The solution at the time, to get
at least some test coverage, was to have tests opt-in to non-intrinsic
debug-info if it was built into LLVM.

Nowadays, non-intrinsic format is the default and has been on for more
than a year, there's no need for this flag to exist.

(I've downgraded the flag from "try" to explicitly requesting
non-intrinsic format in some places, so that we can deal with tests that
are explicitly about non-intrinsic format in their own commit).
2025-03-14 15:50:49 +00:00
Douglas Yung
1d763f3833 Revert "Modify the localCache API to require an explicit commit on CachedFile… (#115331)"
This reverts commit ce9e1d3c15ed6290f1cb07b482939976fa8115cd.

The unittest added in this commit seems to be flaky causing random failure on buildbots:
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13235
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13232
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13228
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13224
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13220
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13210
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13208
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13207
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13202
 - https://lab.llvm.org/buildbot/#/builders/46/builds/13196
and
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14266
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14254
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14250
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14245
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14244
 - https://lab.llvm.org/buildbot/#/builders/180/builds/14226
2025-03-08 23:54:57 +00:00
anjenner
ce9e1d3c15
Modify the localCache API to require an explicit commit on CachedFile… (#115331)
…Stream.

CachedFileStream has previously performed the commit step in its
destructor, but this means its only recourse for error handling is
report_fatal_error. Modify this to add an explicit commit() method, and
call this in the appropriate places with appropriate error handling for
the location.

Currently the destructor of CacheStream gives an assert failure in Debug
builds if commit() was not called. This will help track down any
remaining uses of the API that assume the old destructior behaviour. In
Release builds we fall back to the previous behaviour and call
report_fatal_error if the commit fails.
2025-03-07 17:58:36 +00:00
chrisPyr
71f4c7dabe
[NFC]Make file-local cl::opt global variables static (#126486)
#125983
2025-03-03 13:46:33 +07:00
Nuri Amari
2edd897a42
Make WriteIndexesThinBackend multi threaded (#109847)
We've noticed that for large builds executing thin-link can take on the
order of 10s of minutes. We are only using a single thread to write the
sharded indices and import files for each input bitcode file. While we
need to ensure the index file produced lists modules in a deterministic
order, that doesn't prevent us from executing the rest of the work in
parallel.

In this change we use a thread pool to execute as much of the backend's
work as possible in parallel. In local testing on a machine with 80
cores, this change makes a thin-link for ~100,000 input files run in ~2
minutes. Without this change it takes upwards of 10 minutes.

---------

Co-authored-by: Nuri Amari <nuriamari@fb.com>
2024-10-07 08:16:46 -07:00
Steven Wu
b8c560f159
[CMake] Remove EXPORT_SYMBOLS_FOR_PLUGINS from #102138 (#102396)
Partially remove some of the changes from #102138 as
EXPORT_SYMBOLS_FOR_PLUGINS doesn't work on all the configurations.
2024-08-08 06:00:11 -07:00
Steven Wu
01b488faab
Reapply "[CMake] Fold export_executable_symbols_* into function args. (#101741)" (#102138)
Fix the builds with LLVM_TOOL_LLVM_DRIVER_BUILD enabled.

LLVM_ENABLE_EXPORTED_SYMBOLS_IN_EXECUTABLES is not completely
compatible with export_executable_symbols as the later will be ignored
if the previous is set to NO.

Fix the issue by passing if symbols need to be exported to
llvm_add_exectuable so the link flag can be determined directly
without calling export_executable_symbols_* later.
2024-08-07 09:12:15 -07:00
Steven Wu
f9b69a378c Revert "[CMake] Fold export_executable_symbols_* into function args. (#101741)"
This reverts commit 5c56b46a32a8856a022a54291bc9294068f7ddbd. This break
lld build when using GENERATE_DRIVER.
2024-08-06 06:08:16 -07:00
Steven Wu
5c56b46a32
[CMake] Fold export_executable_symbols_* into function args. (#101741)
`LLVM_ENABLE_EXPORTED_SYMBOLS_IN_EXECUTABLES` is not completely
compatible with `export_executable_symbols` as the later will be ignored
if the previous is set to NO.

 Fix the issue by passing if symbols need to be exported to
`llvm_add_exectuable` so the link flag can be determined directly
without calling `export_executable_symbols_*` later.
2024-08-05 19:08:27 -07:00
Fangrui Song
176ab5e9de [llvm-lto2] Simplify SymbolResolutions loop and avoid expensive std::string copy. NFC 2024-04-26 15:50:44 -07:00
Stephen Tozer
379628d446
[RemoveDIs] Add flag to preserve the debug info format of input IR (#87379)
This patch adds a new flag: `--preserve-input-debuginfo-format`

This flag instructs the tool to not convert the debug info format
(intrinsics/records) of input IR, but to instead determine the format of
the input IR and overwrite the other format-determining flags so that we
process and output the file in the same format that we received it in.
This flag is turned off by llvm-link, llvm-lto, and llvm-lto2, and
should be turned off by any other tool that expects to parse multiple IR
modules and have their debug info formats match.

The motivation for this flag is to allow tools to not convert the debug
info format - verify-uselistorder and llvm-reduce, and any downstream
tools that seek to test or mutate IR as-is, without applying extraneous
modifications to the input. This is a necessary step to using debug
records by default in all (other) LLVM tools.
2024-04-05 14:18:59 +01:00
Orlando Cazalet-Hyams
b3f98dff75
[RemoveDIs] Load into new debug info format by default in llvm-lto and llvm-lto2 (#86271)
Directly load all bitcode into the new debug info format in `llvm-lto`
and `llvm-lto2`. This means that new-mode bitcode no longer round-trips
back to old-mode after parsing, and that old-mode bitcode gets
auto-upgraded to new-mode debug info (which is the current in-memory
default in LLVM).
2024-03-22 13:52:11 +00:00
Jeremy Morse
92eaf036bf
[NFC][RemoveDIs] Remove conditional compilation for RemoveDIs (#81149)
A colleague observes that switching the default value of
LLVM_EXPERIMENTAL_DEBUGINFO_ITERATORS to "On" hasn't flipped the value
in their CMakeCache.txt. This probably means that everyone with an
existing build tree is going to not have support built in, meaning
everyone in LLVM would need to clean+rebuild their worktree when we flip
the switch on... which doesn't sound good.

So instead, just delete the flag and everything it does, making everyone
build and run ~400 lit tests in RemoveDIs mode. None of the buildbots
have had trouble with this, so it Should Be Fine (TM).

(Sending for review as this is changing various comments, and touches
several different areas -- I don't want to get too punchy).
2024-02-08 16:13:22 +00:00
Jeremy Morse
a19629dae7 Reapply 215b8f1e252, reverted in c3f7fb1421e
Turns out I was using DbgMarker::getDbgValueRange rather than the helper
utility in Instruction::getDbgValueRange, which checks for null-ness.
Original commit message follows.

[DebugInfo][RemoveDIs] Convert debug-info modes when loading bitcode (#78967)

As part of eliminating debug-intrinsics in LLVM, we'll shortly be
pushing the conversion from "old" dbg.value mode to "new" DPValue mode
out from when the pass manager runs, to when modules are loaded. This
patch adds that conversion process and some (temporary) options to
llvm-lto{,2} to help test it.

Specifically: now whenever we load a bitcode module, consider a flag of
whether to "upgrade" it into the new debug-info mode, and if we're
lazily materializing functions then do that lazily too. Doing this
exposes an error in the IRLinker/materializer handling of DPValues,
where we need to transfer the debug-info format flag correctly, and in
ValueMapper we need to remap the Values that DPValues point at.

I've added some test coverage in the modified tests; these will be
exercised by our llvm-new-debug-iterators buildbot.

This upgrading of debug-info won't be happening for the llvm18 release,
instead we'll turn it on after the branch date, thenbe push the boundary
of where "new" debug-info starts and ends down into the existing
debug-info upgrade path over the course of the next release.
2024-01-25 18:37:13 +00:00
Jeremy Morse
c3f7fb1421 Revert "[DebugInfo][RemoveDIs] Convert debug-info modes when loading bitcode (#78967)"
This reverts commit 215b8f1e252b4f30cf1b734faa370c0ac4b88659.

Numerous builders exploded from this X_X, for example

  https://lab.llvm.org/buildbot/#/builders/46/builds/62657
2024-01-25 14:18:31 +00:00
Jeremy Morse
215b8f1e25
[DebugInfo][RemoveDIs] Convert debug-info modes when loading bitcode (#78967)
As part of eliminating debug-intrinsics in LLVM, we'll shortly be
pushing the conversion from "old" dbg.value mode to "new" DPValue mode
out from when the pass manager runs, to when modules are loaded. This
patch adds that conversion process and some (temporary) options to
llvm-lto{,2} to help test it.

Specifically: now whenever we load a bitcode module, consider a flag of
whether to "upgrade" it into the new debug-info mode, and if we're
lazily materializing functions then do that lazily too. Doing this
exposes an error in the IRLinker/materializer handling of DPValues,
where we need to transfer the debug-info format flag correctly, and in
ValueMapper we need to remap the Values that DPValues point at.

I've added some test coverage in the modified tests; these will be
exercised by our llvm-new-debug-iterators buildbot.

This upgrading of debug-info won't be happening for the llvm18 release,
instead we'll turn it on after the branch date, thenbe push the boundary
of where "new" debug-info starts and ends down into the existing
debug-info upgrade path over the course of the next release.
2024-01-25 13:27:40 +00:00
Matthew Voss
a1ca3af31e [llvm] A Unified LTO Bitcode Frontend
Here's a high level summary of the changes in this patch. For more
information on rational, see the RFC.
(https://discourse.llvm.org/t/rfc-a-unified-lto-bitcode-frontend/61774).

  - Add config parameter to LTO backend, specifying which LTO mode is
    desired when using unified LTO.
  - Add unified LTO flag to the summary index for efficiency. Unified
    LTO modules can be detected without parsing the module.
  - Make sure that the ModuleID is generated by incorporating more types
    of symbols.

Differential Revision: https://reviews.llvm.org/D123803
2023-07-05 14:53:14 -07:00
Ivan Tadeu Ferreira Antunes Filho
73fd9d310f [lld] Support separate native object file path in --thinlto-prefix-replace
Currently, the --thinlto-prefix-replace="oldpath;newpath" option is used during
distributed ThinLTO thin links to specify the mapping of the input bitcode object
files' directory tree (oldpath) to the directory tree (newpath) used for both:

1) the output files of the thin link itself (the .thinlto.bc index files and the
optional .imports files)
2) the specified object file paths written to the response file given in the
--thinlto-index-only=${response} option, which is used by the final native
link and must match the paths of the native object files that will be
produced by ThinLTO backend compiles.
This patch expands the --thinlto-prefix-replace option to allow a separate directory
tree mapping to be specified for the object file paths written to the response file
(number 2 above). This is important to support builds and build systems where the
same output directory may not be written by multiple build actions (e.g. the thin link
and the ThinLTO backend compiles).

The new format is: --thinlto-prefix-replace="origpath;outpath[;objpath]"

This replaces the origpath directory tree of the thin link input files with
outpath when writing the thin link index and imports outputs (number 1
above). If objpath is specified it replaces origpath of the input files with
objpath when writing the response file (number 2 above), otherwise it
falls back to the old behavior of using outpath for this as well.

Reviewed By: tejohnson, MaskRay

Differential Revision: https://reviews.llvm.org/D144596
2023-04-04 11:24:51 -07:00
Nikita Popov
fb6c1300f2 [LTO] Remove -lto-opaque-pointers flag
Always use the config default of OpaquePointers == true.
2023-01-25 12:18:50 +01:00
Scott Linder
25c0ea2a53 [NFC] Consolidate llvm::CodeGenOpt::Level handling
Add free functions llvm::CodeGenOpt::{getLevel,getID,parseLevel} to
provide common implementations for functionality that has been
duplicated in many places across the codebase.

Differential Revision: https://reviews.llvm.org/D141968
2023-01-23 22:50:49 +00:00
Fangrui Song
d8caac21fb [llvm-lto2] Remove unused include after D123126 2023-01-11 15:33:50 -08:00
Archibald Elliott
f09cf34d00 [Support] Move TargetParsers to new component
This is a fairly large changeset, but it can be broken into a few
pieces:
- `llvm/Support/*TargetParser*` are all moved from the LLVM Support
  component into a new LLVM Component called "TargetParser". This
  potentially enables using tablegen to maintain this information, as
  is shown in https://reviews.llvm.org/D137517. This cannot currently
  be done, as llvm-tblgen relies on LLVM's Support component.
- This also moves two files from Support which use and depend on
  information in the TargetParser:
  - `llvm/Support/Host.{h,cpp}` which contains functions for inspecting
    the current Host machine for info about it, primarily to support
    getting the host triple, but also for `-mcpu=native` support in e.g.
    Clang. This is fairly tightly intertwined with the information in
    `X86TargetParser.h`, so keeping them in the same component makes
    sense.
  - `llvm/ADT/Triple.h` and `llvm/Support/Triple.cpp`, which contains
    the target triple parser and representation. This is very intertwined
    with the Arm target parser, because the arm architecture version
    appears in canonical triples on arm platforms.
- I moved the relevant unittests to their own directory.

And so, we end up with a single component that has all the information
about the following, which to me seems like a unified component:
- Triples that LLVM Knows about
- Architecture names and CPUs that LLVM knows about
- CPU detection logic for LLVM

Given this, I have also moved `RISCVISAInfo.h` into this component, as
it seems to me to be part of that same set of functionality.

If you get link errors in your components after this patch, you likely
need to add TargetParser into LLVM_LINK_COMPONENTS in CMake.

Differential Revision: https://reviews.llvm.org/D137838
2022-12-20 11:05:50 +00:00
Fangrui Song
7d43c3ba51 IR: HotnessThreshold llvm::Optional => std::optional 2022-12-04 19:06:47 +00:00
Zequan Wu
84be92d26f Reland "[LTO][COFF] Use bitcode file names in lto native object file names."
This reverts commit 34108082947c964ae9bbfcd9808f2fd31c0d672f with fixes.
2022-11-22 13:46:42 -08:00
Roman Lebedev
3410808294
Revert "Reland "[LTO][COFF] Use bitcode file names in lto native object file names.""
Breaks build of LLVMgold here:
```
/repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1108:19: error: no matching function for call to 'localCache'
    Cache = check(localCache("ThinLTO", "Thin", options::cache_dir, AddBuffer));
                  ^~~~~~~~~~
/repositories/llvm-project/llvm/include/llvm/Support/Caching.h:72:21: note: candidate function not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1102:20)' to 'llvm::AddBufferFn' (aka 'function<void (unsigned int, const llvm::Twine &, std::unique_ptr<MemoryBuffer>)>') for 4th argument
Expected<FileCache> localCache(
                    ^
/repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1110:18: error: no viable conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'llvm::AddStreamFn' (aka 'function<Expected<std::unique_ptr<CachedFileStream>> (unsigned int, const llvm::Twine &)>')
  check(Lto->run(AddStream, Cache));
                 ^~~~~~~~~
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:375:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::nullptr_t' for 1st argument
      function(nullptr_t) noexcept
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:386:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'const std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream>> (unsigned int, const llvm::Twine &)> &' for 1st argument
      function(const function& __x)
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:404:7: note: candidate constructor not viable: no known conversion from '(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20)' to 'std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream>> (unsigned int, const llvm::Twine &)> &&' for 1st argument
      function(function&& __x) noexcept
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:435:2: note: candidate template ignored: requirement '_Callable<(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &, (lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20), std::__invoke_result<(lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &, unsigned int, const llvm::Twine &>>::value' was not satisfied [with _Functor = (lambda at /repositories/llvm-project/llvm/tools/gold/gold-plugin.cpp:1094:20) &]
        function(_Functor&& __f)
        ^
/repositories/llvm-project/llvm/include/llvm/LTO/LTO.h:278:25: note: passing argument to parameter 'AddStream' here
  Error run(AddStreamFn AddStream, FileCache Cache = nullptr);
                        ^

```

This reverts commit 387620aa8cea33174b6c1fb80c1af713fee732ac.
2022-11-23 00:40:12 +03:00
Zequan Wu
387620aa8c Reland "[LTO][COFF] Use bitcode file names in lto native object file names."
This reverts commit eef5405f74ae208e3e2eb7daacecac923d7338f2.
2022-11-22 11:26:18 -08:00
Zequan Wu
eef5405f74 Revert "[LTO][COFF] Use bitcode file names in lto native object file names."
This reverts commit 531ed6d5aa65f41c6dfe2e74905d5c6c88fc95a7.
2022-11-22 10:55:05 -08:00
Zequan Wu
531ed6d5aa [LTO][COFF] Use bitcode file names in lto native object file names.
Currently the lto native object files have names like main.exe.lto.1.obj. In
PDB, those names are used as names for each compiland. Microsoft’s tool
SizeBench uses those names to present to users the size of each object files.
So, names like main.exe.lto.1.obj is not user friendly.

This patch makes the lto native object file names more readable by using
the bitcode file names as part of the file names. For example, if the input
bitcode file has path like "path/to/foo.obj", its corresponding lto native
object file path would be "path/to/main.exe.lto.foo.obj". Since the lto native
object file name only bothers PDB, this patch only changes the lld-linker's
behavior.

Reviewed By: tejohnson, MaskRay, #lld-macho

Differential Revision: https://reviews.llvm.org/D137217
2022-11-22 10:19:58 -08:00
Jin Xin Ng
65001f5777
[LTO][ELF] Add selective --save-temps= option
Allows specific “temps” to be saved, instead of the current all-or-nothing nature of --save-temps. Multiple of these “temps” can be saved by specifying the argument multiple times.

Differential Revision: https://reviews.llvm.org/D127778
2022-07-06 10:06:18 -07:00
Fangrui Song
fd25a0aa41 [llvm-lto2] Remove unneeded cl::init(false). NFC 2022-07-01 14:35:36 -07:00
Jin Xin Ng
f702c7bb9e
[ThinLTO][test] Add tests for emitting files in-process
Completes D127777 by adding llvm-side tests for emitting index and imports files from in-process ThinLTO

Differential Revision: https://reviews.llvm.org/D128771
2022-06-29 14:42:45 -07:00
Kazu Hirata
7a47ee51a1 [llvm] Don't use Optional::getValue (NFC) 2022-06-20 22:45:45 -07:00
Fangrui Song
95a134254a Remove unneeded cl::ZeroOrMore for cl::opt/cl::list options 2022-06-05 01:07:51 -07:00
Fangrui Song
36c7d79dc4 Remove unneeded cl::ZeroOrMore for cl::opt options
Similar to 557efc9a8b68628c2c944678c6471dac30ed9e8e.
This commit handles options where cl::ZeroOrMore is more than one line below
cl::opt.
2022-06-04 00:10:42 -07:00
Matthias Braun
850d53a197 LTO: Decide upfront whether to use opaque/non-opaque pointer types
LTO code may end up mixing bitcode files from various sources varying in
their use of opaque pointer types. The current strategy to decide
between opaque / typed pointers upon the first bitcode file loaded does
not work here, since we could be loading a non-opaque bitcode file first
and would then be unable to load any files with opaque pointer types
later.

So for LTO this:
- Adds an `lto::Config::OpaquePointer` option and enforces an upfront
  decision between the two modes.
- Adds `-opaque-pointers`/`-no-opaque-pointers` options to the gold
  plugin; disabled by default.
- `--opaque-pointers`/`--no-opaque-pointers` options with
  `-plugin-opt=-opaque-pointers`/`-plugin-opt=-no-opaque-pointers`
  aliases to lld; disabled by default.
- Adds an `-lto-opaque-pointers` option to the `llvm-lto2` tool.
- Changes the clang driver to pass `-plugin-opt=-opaque-pointers` to
  the linker in LTO modes when clang was configured with opaque
  pointers enabled by default.

This fixes https://github.com/llvm/llvm-project/issues/55377

Differential Revision: https://reviews.llvm.org/D125847
2022-06-01 18:05:53 -07:00
Nikita Popov
2121dc5b15 [llvm-lto] Remove support for legacy pass manager
This removes support for the legacy pass manager in llvm-lto and
llvm-lto2. In this case I've dropped the use-new-pm option entirely,
as I don't think this is considered part of the public interface.

This also makes -debug-pass-manager work with llvm-lto, because
that was needed to migrate some tests to NewPM.

Differential Revision: https://reviews.llvm.org/D123376
2022-04-11 09:40:17 +02:00
Nikita Popov
ed4e6e0398 [cmake] Remove LLVM_ENABLE_NEW_PASS_MANAGER cmake option
Or rather, error out if it is set to something other than ON. This
removes the ability to enable the legacy pass manager by default,
but does not remove the ability to explicitly enable it through
various flags like -flegacy-pass-manager or -enable-new-pm=0.

I checked, and our test suite definitely doesn't pass with
LLVM_ENABLE_NEW_PASS_MANAGER=OFF anymore.

Differential Revision: https://reviews.llvm.org/D123126
2022-04-06 09:52:21 +02:00
Alexandre Ganea
a282ea4898 Reland - [CodeView] Emit S_OBJNAME record
Reland integrates build fixes & further review suggestions.

Thanks to @zturner for the initial S_OBJNAME patch!

Differential Revision: https://reviews.llvm.org/D43002
2021-12-21 19:02:14 -05:00
Alexandre Ganea
5bb5142e80 Revert [CodeView] Emit S_OBJNAME record
Also revert all subsequent fixes:
- abd1cbf5e543f0f114d2742e109ead7d7ddbf9c4 [Clang] Disable debug-info-objname.cpp test on Unix until I sort out the issue.
- 00ec441253048f5e30540ea26bb0a28c42a5fc18 [Clang] debug-info-objname.cpp test: explictly encode a x86 target when using %clang_cl to avoid falling back to a native CPU triple.
- cd407f6e52b09cce2bef24c74b7f36fedc94991b [Clang] Fix build by restricting debug-info-objname.cpp test to x86.
2021-12-21 19:02:14 -05:00
Alexandre Ganea
f44e3fbadd [CodeView] Emit S_OBJNAME record
Thanks to @zturner for the initial patch!

Differential Revision: https://reviews.llvm.org/D43002
2021-12-21 09:26:36 -05:00
Mircea Trofin
4fed39ddee [ThinLTO] Fix nondeterministic exit on error.
In the multi-threaded case, if a thread hits an error, we mimick
LLVMContext's behavior of reporting the error and exit-ing. However,
this doesn't cleanly join the other threads, so depending on how fast
the process exits, other threads may report 'terminate called without an
active exception'.

To avoid this non-determinsim, and without introducing a more complicated
design, we just report the error, but not exit early. We do track whether
we hit errors and exit(1) after joining.

Differential Revision: https://reviews.llvm.org/D115574
2021-12-13 10:08:39 -08:00
Noah Shutty
d788c44f5c [Support] Improve Caching conformance with Support library behavior
This diff makes several amendments to the local file caching mechanism
which was migrated from ThinLTO to Support in
rGe678c51177102845c93529d457b020f969125373 in response to follow-up
discussion on that commit.

Patch By: noajshu

Differential Revision: https://reviews.llvm.org/D113080
2021-11-04 13:00:44 -07:00
Noah Shutty
e678c51177 [Support][ThinLTO] Move ThinLTO caching to LLVM Support library
We would like to move ThinLTO’s battle-tested file caching mechanism to
the LLVM Support library so that we can use it elsewhere in LLVM.

Patch By: noajshu

Differential Revision: https://reviews.llvm.org/D111371
2021-10-18 18:57:25 -07:00