ArrayRef now has a new constructor that takes a parameter whose type
has data() and size(). This patch migrates:
ArrayRef<T>(X.data(), X.size()
to:
ArrayRef<T>(X)
Currently ROCm 6.4.0 only recognize spirv64-amd-amdhsa- in bundle ID.
spirv64-amd-amdhsa-unknown causes all HIP apps compiled for amdgcnspirv
to fail.
Previously we fixed a similar issue for
amdgcn-amd-amdhsa-unknown. This patch extends that to
spirv64-amd-amdhsa-unknown.
In COMGR we hash the header of compressed bundles. For this we take the
first bytes of the buffer (according to the maximum header size) and
hash them.
To have a more stable API, and to be able to pick only the hash field (which is
the only one we are actually interested in) of the header, we propose
a version independent header version that is common to all versions.
The current parsing logic for the target string assumes it follows the
format `<kind>-<triple>-<target id>:<feature>`, such as
`hipv4-amdgcn-amd-amdhsa-gfx1030:+xnack`.
Specifically, it assumes that `<target id>` does not contain any `-`,
relying on `rsplit` for parsing.
However, this assumption breaks for AMDGPU's generic targets, which may
contain one or more `-`, such as `gfx10-3-generic` or `gfx12-generic`.
As a result, the existing approach using `rstrip` is no longer reliable.
This patch reworks the parsing logic to handle target strings more
robustly, including support for generic targets.
The bundler now strictly requires a 4-field target triple.
Additionally, a new Python helper function has been added to `config.py`
to normalize the target triple into the 4-field format when it is not,
ensuring tests pass reliably.
Added initial support for version 3 of the compressed offload bundle
format, which uses 64-bit fields for Total File Size and Uncompressed
Binary Size. This enables support for files larger than 4GB. The support
is currently experimental and can be enabled by setting the environment
variable `COMPRESSED_BUNDLE_FORMAT_VERSION=3`.
This is intended to help with flang `-ftime-report` support:
- #107270.
With this change, I was able to cherry-pick #107270, uncomment
`llvm::TimePassesIsEnabled = true;` and compile with `-ftime-report`.
I also noticed that `clang/lib/Driver/OffloadBundler.cpp` was statically
constructing a `TimerGroup` and changed it to lazily construct via
ManagedStatic.
The distinction between the hip and hipv4 offload kinds is historically
based. Originally, these designations might have indicated different
versions of the code object ABI (Application Binary Interface). However,
as the system has evolved, the ABI version is now embedded directly
within the code object itself, making these historical distinctions
irrelevant during the unbundling process. Consequently, hip and hipv4
are treated as compatible in current implementations, facilitating
interchangeable handling of code objects without differentiation based
on offload kind. This change streamlines code management within the
ecosystem.
__hipRegisterFatBinary only accepts one pointer argument. It is expected
to get the fat binary size from the header.
This patch adds a file size field to the header of the compressed
bundle.
Added --offload-compression-level= option to clang and
-compression-level=
option to clang-offload-bundler for controlling compression level.
Added support of long distance matching (LDM) for llvm::zstd which is
off
by default. Enable it for clang-offload-bundler by default since it
improves compression rate in general.
Change default compression level to 3 for zstd for clang-offload-bundler
since it works well for bundle entry size from 1KB to 32MB, which should
cover most of the clang-offload-bundler usage. Users can still specify
compression level by -compression-level= option if necessary.
When unbundling an archive, need to save the content of each object file
to a temporary file before passing it to llvm-objcopy, instead of
passing the original input archive file to llvm-objcopy.
Also allows extracting host bundles for archives.
Fixes: https://github.com/llvm/llvm-project/issues/83509
`-fgpu-rdc` mode allows device functions call device functions in
different TU. However, currently all device objects have to be linked
together since only one fat binary is supported. This is time consuming
for AMDGPU backend since it only supports LTO.
There are use cases that objects can be divided into groups in which
device functions are self-contained but host functions are not. It is
desirable to link/optimize/codegen the device code and generate a fatbin
for each group, whereas partially link the host code with `ld -r` or
generate a static library by using the `--emit-static-lib` option of
clang. This avoids linking all device code together, therefore decreases
the linking time for `-fgpu-rdc`.
Previously, clang emits an external symbol `__hip_fatbin` for all
objects for `-fgpu-rdc`. With this patch, clang emits an unique external
symbol `__hip_fatbin_{cuid}` for the fat binary for each object. When a
group of objects are linked together to generate a fatbin, the symbols
are merged by alias and point to the same fat binary. Each group has its
own fat binary. One executable or shared library can have multiple fat
binaries. Device linking is done for undefined fab binary symbols only
to avoid repeated linking. `__hip_gpubin_handle` is also uniquefied and
merged to avoid repeated registering. Symbol `__hip_cuid_{cuid}` is
introduced to facilitate debugging and tooling.
Fixes: https://github.com/llvm/llvm-project/issues/77018
Note that llvm::support::endianness has been renamed to
llvm::endianness while becoming an enum class as opposed to an enum.
This patch replaces llvm::support::{big,little,native} with
llvm::endianness::{big,little,native}.
Original PR: https://github.com/llvm/llvm-project/pull/67162
The commit was reverted due to UB detected by santizer:
https://lab.llvm.org/buildbot/#/builders/238/builds/5955
clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error:
load of misaligned address 0xaaaae2d90e7c for type
'const uint64_t' (aka 'const unsigned long'), which
requires 8 byte alignment
It was fixed by using memcpy instead of dereferencing int*
casted from unaligned char*.
This reverts commit a1e81d2ead02e041471ec2299d7382f80c4dbba6.
Revert "Fix test hip-offload-compress-zlib.hip"
This reverts commit ba01ce60665848478ba4e76190907153a8c26fe9.
Revert due to sanity fail at
https://lab.llvm.org/buildbot/#/builders/5/builds/37188https://lab.llvm.org/buildbot/#/builders/238/builds/5955
/b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25: runtime error: load of misaligned address 0xaaaae2d90e7c for type 'const uint64_t' (aka 'const unsigned long'), which requires 8 byte alignment
0xaaaae2d90e7c: note: pointer points here
bc 00 00 00 94 dc 29 9a 89 fb ca 2b 78 9c 8b 8f 77 f6 71 f4 73 8f f7 77 73 f3 f1 77 74 89 77 0a
^
#0 0xaaaaba125f70 in clang::CompressedOffloadBundle::decompress(llvm::MemoryBuffer const&, bool) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1012:25
#1 0xaaaaba126150 in clang::OffloadBundler::ListBundleIDsInFile(llvm::StringRef, clang::OffloadBundlerConfig const&) /b/sanitizer-aarch64-linux-bootstrap-ubsan/build/llvm-project/clang/lib/Driver/OffloadBundler.cpp:1089:7
Will reland after fixing it.
Add option -f[no-]offload-compress to clang to enable/disable
compression of device binary for HIP. By default it is disabled.
Add option -compress to clang-offload-bundler to enable compression of
offload bundle. By default it is disabled.
When enabled, zstd or zlib is used for compression when available.
When disabled, it is NFC compared to previous behavior. The same offload
bundle format is used as before.
Clang-offload-bundler automatically detects whether the input file to be
unbundled is compressed and the compression method and decompress if
necessary.
Summary:
llvm-ar is symlinked as llvm-ranlib and will act as ranlib when invoked in that mode. llvm-ar since [[ 4f2cfbe531 | compiler/llvm-project@4f2cfbe ]] supports the -X options, but doesn't seem to accept them when running as llvm-ranlib.
In AIX OS , according to https://www.ibm.com/docs/en/aix/7.2?topic=r-ranlib-command
-X mode Specifies the type of object file ranlib should examine. The mode must be one of the following:
32
Processes only 32-bit object files
64
Processes only 64-bit object files
32_64, any
Processes both 32-bit and 64-bit object files
The default is to process 32-bit object files (ignore 64-bit objects). The mode can also be set with the OBJECT_MODE environment variable. For example, OBJECT_MODE=64 causes ranlib to process any 64-bit objects and ignore 32-bit objects. The -X flag overrides the OBJECT_MODE variable.
Reviewers: James Henderson, MaskRay, Stephen Peckham
Differential Revision: https://reviews.llvm.org/D142660
This patch migrates uses of StringRef::{starts,ends}with_insensitive
to StringRef::{starts,ends}_with_insensitive so that we can use names
similar to those used in std::string_view.
Note that the llvm/ directory has migrated in commit
6c3ea866e93003e16fc55d3b5cedd3bc371d1fde.
I'll post a separate patch to deprecate
StringRef::{starts,ends}with_insensitive.
Differential Revision: https://reviews.llvm.org/D150506
The bundler accepts both of the following for the --target option:
hip-amdgcn-amd-amdhsa-gfx900 (no env field)
hip-amdgcn-amd-amdhsa--gfx900 (blank env field)
The environment field is defined as optional for Triples
in Triple.h. However, in this patch we update the bundler to
internally standardize to include the env field. While users
aren't required to specify an env field when listing targets on
the commandline, bundles generated by the offload-bundler will
include the ABI field.
This standardization simplifies things for APIs that deal with
bundles generated by the clang-offload-bundler tool.
Differential Revision: https://reviews.llvm.org/D145770
The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.
so that clang can recognize it and handle it automatically
without -x hip-cpp-output.
Reviewed by: Artem Belevich
Differential Revision: https://reviews.llvm.org/D141437
In HIP a library is usually compiled with default target ID e.g. gfx906 so that
it can be used in all GPU configurations. The bitcode is saved in bundled
bitcode with gfx906 in entry ID.
In runtime compilation, a HIP program is compiled with a target ID matching
the GPU configuration, e.g. gfx906:xnack-. This program needs to link with
a library bundled bitcode with target ID gfx906.
For example:
clang --offload-arch=gfx906 -o lib.o lib.hip
clang --offload-arch=gfx906:xnack- program.hip lib.o
This common use case requires that clang-offlod-bundler to be able to extract
entry with compatible target ID, e.g. extracting an gfx906 entry when requesting
gfx906:xnack-.
Currently clang-offload-bundler only allow extracting entry with exact match
of target ID. This patch relaxes that so that it can extract entries with compatible
target ID.
Reviewed by: Artem Belevich, Saiyedul Islam
Differential Revision: https://reviews.llvm.org/D134546
Lifting the core functionalities of the clang-offload-bundler into a
user-facing library/API. This will allow online and JIT compilers to
bundle and unbundle files without spawning a new process.
This patch lifts the classes and functions used to implement
the clang-offload-bundler into a separate OffloadBundler.cpp,
and defines three top-level API functions in OfflaodBundler.h.
BundleFiles()
UnbundleFiles()
UnbundleArchives()
This patch also introduces a Config class that locally stores the
previously global cl::opt options and arrays to allow users to call
the APIs in a multi-threaded context, and introduces an
OffloadBundler class to encapsulate the top-level API functions.
We also lift the BundlerExecutable variable, which is specific
to the clang-offload-bundler tool, from the API, and replace
its use with an ObjcopyPath variable. This variable must be set
in order to internally call llvm-objcopy.
Finally, we move the API files from
clang/tools/clang-offload-bundler into clang/lib/Driver and
clang/include/clang/Driver.
Differential Revision: https://reviews.llvm.org/D129873
Lifting the core functionalities of the clang-offload-bundler into a
user-facing library/API. This will allow online and JIT compilers to
bundle and unbundle files without spawning a new process.
This patch lifts the classes and functions used to implement
the clang-offload-bundler into a separate OffloadBundler.cpp,
and defines three top-level API functions in OfflaodBundler.h.
BundleFiles()
UnbundleFiles()
UnbundleArchives()
This patch also introduces a Config class that locally stores the
previously global cl::opt options and arrays to allow users to call
the APIs in a multi-threaded context, and introduces an
OffloadBundler class to encapsulate the top-level API functions.
We also lift the BundlerExecutable variable, which is specific
to the clang-offload-bundler tool, from the API, and replace
its use with an ObjcopyPath variable. This variable must be set
in order to internally call llvm-objcopy.
Finally, we move the API files from
clang/tools/clang-offload-bundler into clang/lib/Driver and
clang/include/clang/Driver.
Differential Revision: https://reviews.llvm.org/D129873
