These allow performing optimizations that reduce I/O and disk space
consumption. For example, when applicable, a file can be cloned directly
into the database directory, instead of needing to load it in memory and
then copy its contents into a new file.
These APIs are then used to optimize importing data from an upstream DB
by using file cloning where applicable.
Adds a CAS logging interface to track mutations of the CAS and action
cache database files. This is designed to aid in the debugging of
data corruption and other issues with the low-level on-disk
representation.
To enable logging, set the environment variable LLVM_CAS_LOG to 1 or 2.
Level 2 includes logging of all allocations inside the index, action
cache, and datapool, which increases the overhead. The log file is
placed inside the top-level directory of the unified cache. For now it
is a human-readable textual format with one line per entry.
This PR bypasses sandbox in the on-disk CAS implementation to match the
downstream patch: https://github.com/swiftlang/llvm-project/pull/12022
This isn't strictly necessary right now, since Clang doesn't integrate
with CAS upstream, but it'll make the future integration easier.
Check that action cache references point to valid CAS objects by
ensuring they are contained within the corresponding CAS and also that
the offsets match. This prevents accidentally referencing "dead" index
records that were not properly flushed to disk, which can lead to the
action cache pointing to the wrong data or to garbage data.
rdar://126642956
Building with GCC I got:
```
<...>/OnDiskGraphDB.cpp:624:18: warning: ‘static {anonymous}::DataRecordHandle {anonymous}::DataRecordHandle::construct(char*, const {anonymous}::DataRecordHandle::Input&)’ defined but not used [-Wunused-function]
624 | DataRecordHandle DataRecordHandle::construct(char *Mem, const Input &I) {
| ^~~~~~~~~~~~~~~~
<...>/OnDiskGraphDB.cpp:456:1: warning: ‘static {anonymous}::DataRecordHandle {anonymous}::DataRecordHandle::create(llvm::function_ref<char*(long unsigned int)>, const {anonymous}::DataRecordHandle::Input&)’ defined but not used [-Wunused-function]
456 | DataRecordHandle::create(function_ref<char *(size_t Size)> Alloc,
| ^~~~~~~~~~~~~~~~
```
These implement parts of a class that is defined in an anonymous
namespace. All llvm tests passed with them removed.
Add a new abstraction layer UnifiedOnDiskCache that adds new functions
of disk space management and data validation that builds on top of
OnDiskGraphDB and OnDiskKeyValueDB.
Build upon UnifiedOnDiskCache, it is OnDiskCAS that implements
ObjectStore and ActionCache interface for LLVM tools to interact with
CAS storage.
Add OnDiskGraphDB and OnDiskKeyValueDB that can be used to implement
ObjectStore and ActionCache respectively. Those are on-disk persistent
storage that build upon OnDiskTrieHashMap and implements key functions
that are required by LLVMCAS interfaces.
This abstraction layer defines how the objects are hashed and stored on
disk. OnDiskKeyValueDB is a basic OnDiskTrieHashMap while OnDiskGraphDB
also defines:
* How objects of various size are store on disk and are referenced by
the trie nodes.
* How to store the references from one stored object to another object
that is referenced.
In addition to basic APIs for ObjectStore and ActionCache, other
advances database configuration features can be implemented in this
layer without exposing to the users of the LLVMCAS interface. For
example, OnDiskGraphDB has a faulty in function to fetch data from an
upstream OnDiskGraphDB if the data is missing.
