Lang Hames 6fa8657a62
[ORC] Refactor visit-members in StaticLibraryDefinitionGenerator. (#141546)
This refactor was motivated by two bugs identified in out-of-tree
builds:

1. Some implementations of the VisitMembersFunction type (often used to	
implement special loading semantics, e.g. -all_load or -ObjC) were assuming
that buffers for archive members were null-terminated, which they are not in
general. This was triggering occasional assertions.

2. Archives may include multiple members with the same file name, e.g.
when constructed by appending files with the same name:
  % llvm-ar crs libfoo.a foo.o
  % llvm-ar q libfoo.a foo.o
  % llvm-ar t libfoo.a foo.o
  foo.o

   While confusing, these members may be safe to link (provided that they're
   individually valid and don't define duplicate symbols). In ORC however, the
   archive member name may be used to construct an ORC initializer symbol,
   which must also be unique. In that case the duplicate member names lead to a
   duplicate definition error even if the members define unrelated symbols.

In addition to these bugs, StaticLibraryDefinitionGenerator had grown a
collection of all member buffers (ObjectFilesMap), a BumpPtrAllocator
that was redundantly storing synthesized archive member names (these are
copied into the MemoryBuffers created for each Object, but were never
freed in the allocator), and a set of COFF-specific import files.

To fix the bugs above and simplify StaticLibraryDefinitionGenerator this
patch makes the following changes:

1. StaticLibraryDefinitionGenerator::VisitMembersFunction is generalized
   to take a reference to the containing archive, and the index of the
   member within the archive. It now returns an Expected<bool> indicating
   whether the member visited should be treated as loadable, not loadable,
   or as invalidating the entire archive.
2. A static StaticLibraryDefinitionGenerator::createMemberBuffer method
   is added which creates MemoryBuffers with unique names of the form
   `<archive-name>[<index>](<member-name>)`. This defers construction of
   member names until they're loaded, allowing the BumpPtrAllocator (with
   its redundant name storage) to be removed.
3. The ObjectFilesMap (symbol name -> memory-buffer-ref) is replaced
   with a SymbolToMemberIndexMap (symbol name -> index) which should be
   smaller and faster to construct.
4. The 'loadability' result from VisitMemberFunctions is now taken into
   consideration when building the SymbolToMemberIndexMap so that members
   that have already been loaded / filtered out can be skipped, and do not
   take up any ongoing space.
5. The COFF ImportedDynamicLibraries member is moved out into the
   COFFImportFileScanner utility, which can be used as a
   VisitMemberFunction.

This fixes the bugs described above; and should lower memory consumption
slightly, especially for archives with many files and / or symbol where
most files are eventually loaded.
2025-05-27 20:58:53 +10:00

163 lines
6.4 KiB
C++

//===---- llvm-jitlink.h - Session and format-specific decls ----*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// llvm-jitlink Session class and tool utilities.
//
//===----------------------------------------------------------------------===//
#ifndef LLVM_TOOLS_LLVM_JITLINK_LLVM_JITLINK_H
#define LLVM_TOOLS_LLVM_JITLINK_LLVM_JITLINK_H
#include "llvm/ADT/StringSet.h"
#include "llvm/ExecutionEngine/Orc/COFF.h"
#include "llvm/ExecutionEngine/Orc/Core.h"
#include "llvm/ExecutionEngine/Orc/ExecutorProcessControl.h"
#include "llvm/ExecutionEngine/Orc/LazyObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/Orc/LazyReexports.h"
#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/Orc/RedirectionManager.h"
#include "llvm/ExecutionEngine/Orc/SimpleRemoteEPC.h"
#include "llvm/ExecutionEngine/RuntimeDyldChecker.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/Regex.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/TargetParser/SubtargetFeature.h"
#include "llvm/TargetParser/Triple.h"
namespace llvm {
struct Session {
struct LazyLinkingSupport {
LazyLinkingSupport(
std::unique_ptr<orc::RedirectableSymbolManager> RSMgr,
std::shared_ptr<orc::SimpleLazyReexportsSpeculator> Speculator,
std::unique_ptr<orc::LazyReexportsManager> LRMgr,
orc::ObjectLinkingLayer &ObjLinkingLayer)
: RSMgr(std::move(RSMgr)), Speculator(std::move(Speculator)),
LRMgr(std::move(LRMgr)),
LazyObjLinkingLayer(ObjLinkingLayer, *this->LRMgr) {}
std::unique_ptr<orc::RedirectableSymbolManager> RSMgr;
std::shared_ptr<orc::SimpleLazyReexportsSpeculator> Speculator;
std::unique_ptr<orc::LazyReexportsManager> LRMgr;
orc::LazyObjectLinkingLayer LazyObjLinkingLayer;
};
orc::ExecutionSession ES;
orc::JITDylib *MainJD = nullptr;
orc::JITDylib *ProcessSymsJD = nullptr;
orc::JITDylib *PlatformJD = nullptr;
orc::ObjectLinkingLayer ObjLayer;
std::unique_ptr<LazyLinkingSupport> LazyLinking;
orc::JITDylibSearchOrder JDSearchOrder;
SubtargetFeatures Features;
std::vector<std::pair<std::string, orc::SymbolStringPtr>> LazyFnExecOrder;
~Session();
static Expected<std::unique_ptr<Session>> Create(Triple TT,
SubtargetFeatures Features);
void dumpSessionInfo(raw_ostream &OS);
void modifyPassConfig(jitlink::LinkGraph &G,
jitlink::PassConfiguration &PassConfig);
/// For -check: wait for all files that are referenced (transitively) from
/// the entry point *file* to be linked. (ORC's usual dependence tracking is
/// to fine-grained here: a lookup of the main symbol will return as soon as
/// all reachable symbols have been linked, but testcases may want to
/// inspect side-effects in unreachable symbols)..
void waitForFilesLinkedFromEntryPointFile() {
std::unique_lock<std::mutex> Lock(M);
return ActiveLinksCV.wait(Lock, [this]() { return ActiveLinks == 0; });
}
using MemoryRegionInfo = RuntimeDyldChecker::MemoryRegionInfo;
struct FileInfo {
StringMap<MemoryRegionInfo> SectionInfos;
StringMap<SmallVector<MemoryRegionInfo, 1>> StubInfos;
StringMap<MemoryRegionInfo> GOTEntryInfos;
using Symbol = jitlink::Symbol;
using LinkGraph = jitlink::LinkGraph;
using GetSymbolTargetFunction =
unique_function<Expected<Symbol &>(LinkGraph &G, jitlink::Block &)>;
Error registerGOTEntry(LinkGraph &G, Symbol &Sym,
GetSymbolTargetFunction GetSymbolTarget);
Error registerStubEntry(LinkGraph &G, Symbol &Sym,
GetSymbolTargetFunction GetSymbolTarget);
Error registerMultiStubEntry(LinkGraph &G, Symbol &Sym,
GetSymbolTargetFunction GetSymbolTarget);
};
using DynLibJDMap = std::map<std::string, orc::JITDylib *, std::less<>>;
using SymbolInfoMap = DenseMap<orc::SymbolStringPtr, MemoryRegionInfo>;
using FileInfoMap = StringMap<FileInfo>;
Expected<orc::JITDylib *> getOrLoadDynamicLibrary(StringRef LibPath);
Error loadAndLinkDynamicLibrary(orc::JITDylib &JD, StringRef LibPath);
orc::ObjectLayer &getLinkLayer(bool Lazy) {
assert((!Lazy || LazyLinking) &&
"Lazy linking requested but not available");
return Lazy ? static_cast<orc::ObjectLayer &>(
LazyLinking->LazyObjLinkingLayer)
: static_cast<orc::ObjectLayer &>(ObjLayer);
}
Expected<FileInfo &> findFileInfo(StringRef FileName);
Expected<MemoryRegionInfo &> findSectionInfo(StringRef FileName,
StringRef SectionName);
Expected<MemoryRegionInfo &> findStubInfo(StringRef FileName,
StringRef TargetName,
StringRef KindNameFilter);
Expected<MemoryRegionInfo &> findGOTEntryInfo(StringRef FileName,
StringRef TargetName);
bool isSymbolRegistered(const orc::SymbolStringPtr &Name);
Expected<MemoryRegionInfo &> findSymbolInfo(const orc::SymbolStringPtr &Name,
Twine ErrorMsgStem);
DynLibJDMap DynLibJDs;
std::mutex M;
std::condition_variable ActiveLinksCV;
size_t ActiveLinks = 0;
SymbolInfoMap SymbolInfos;
FileInfoMap FileInfos;
StringSet<> HarnessFiles;
StringSet<> HarnessExternals;
StringSet<> HarnessDefinitions;
DenseMap<StringRef, StringRef> CanonicalWeakDefs;
StringSet<> HiddenArchives;
std::optional<Regex> ShowGraphsRegex;
private:
Session(std::unique_ptr<orc::ExecutorProcessControl> EPC, Error &Err);
};
/// Record symbols, GOT entries, stubs, and sections for ELF file.
Error registerELFGraphInfo(Session &S, jitlink::LinkGraph &G);
/// Record symbols, GOT entries, stubs, and sections for MachO file.
Error registerMachOGraphInfo(Session &S, jitlink::LinkGraph &G);
/// Record symbols, GOT entries, stubs, and sections for COFF file.
Error registerCOFFGraphInfo(Session &S, jitlink::LinkGraph &G);
/// Adds a statistics gathering plugin if any stats options are used.
void enableStatistics(Session &S, bool UsingOrcRuntime);
} // end namespace llvm
#endif // LLVM_TOOLS_LLVM_JITLINK_LLVM_JITLINK_H