This patch adds two versions of memcpy optimized for architectures where unaligned accesses are either illegal or extremely slow.
It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well.
Here is the before / after output of `libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memcpy` on a quad core Linux starfive RISCV 64 board running at 1.5GHz.
Before:
```
Run on (4 X 1500 MHz CPU s)
CPU Caches:
L1 Instruction 32 KiB (x4)
L1 Data 32 KiB (x4)
L2 Unified 2048 KiB (x1)
------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------
BM_Memcpy/0/0 474 ns 474 ns 1483776 bytes_per_cycle=0.243492/s bytes_per_second=348.318M/s items_per_second=2.11097M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0 210 ns 209 ns 3649536 bytes_per_cycle=0.233819/s bytes_per_second=334.481M/s items_per_second=4.77519M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0 1814 ns 1814 ns 396288 bytes_per_cycle=0.247899/s bytes_per_second=354.622M/s items_per_second=551.402k/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0 89.3 ns 89.2 ns 7459840 bytes_per_cycle=0.217415/s bytes_per_second=311.014M/s items_per_second=11.2071M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0 134 ns 134 ns 3815424 bytes_per_cycle=0.226584/s bytes_per_second=324.131M/s items_per_second=7.44567M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0 52.8 ns 52.6 ns 11001856 bytes_per_cycle=0.194893/s bytes_per_second=278.797M/s items_per_second=19.0284M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0 180 ns 180 ns 4101120 bytes_per_cycle=0.231884/s bytes_per_second=331.713M/s items_per_second=5.55957M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0 195 ns 195 ns 3906560 bytes_per_cycle=0.232951/s bytes_per_second=333.239M/s items_per_second=5.1217M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0 152 ns 152 ns 4789248 bytes_per_cycle=0.227507/s bytes_per_second=325.452M/s items_per_second=6.58187M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0 6036 ns 6033 ns 118784 bytes_per_cycle=0.249158/s bytes_per_second=356.423M/s items_per_second=165.75k/s __llvm_libc::memcpy,uniform 384 to 4096
```
After:
```
BM_Memcpy/0/0 126 ns 126 ns 5770240 bytes_per_cycle=1.04707/s bytes_per_second=1.46273G/s items_per_second=7.9385M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0 75.1 ns 75.0 ns 10204160 bytes_per_cycle=0.691143/s bytes_per_second=988.687M/s items_per_second=13.3289M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0 333 ns 333 ns 2174976 bytes_per_cycle=1.39297/s bytes_per_second=1.94596G/s items_per_second=3.00002M/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0 49.6 ns 49.5 ns 16092160 bytes_per_cycle=0.710161/s bytes_per_second=1015.89M/s items_per_second=20.1844M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0 57.7 ns 57.7 ns 11213824 bytes_per_cycle=0.561557/s bytes_per_second=803.314M/s items_per_second=17.3228M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0 48.0 ns 47.9 ns 16437248 bytes_per_cycle=0.346708/s bytes_per_second=495.97M/s items_per_second=20.8571M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0 67.5 ns 67.5 ns 10616832 bytes_per_cycle=0.614173/s bytes_per_second=878.582M/s items_per_second=14.8142M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0 84.7 ns 84.6 ns 10480640 bytes_per_cycle=0.819077/s bytes_per_second=1.14424G/s items_per_second=11.8174M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0 61.7 ns 61.6 ns 11191296 bytes_per_cycle=0.550078/s bytes_per_second=786.893M/s items_per_second=16.2279M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0 981 ns 981 ns 703488 bytes_per_cycle=1.52333/s bytes_per_second=2.12807G/s items_per_second=1019.81k/s __llvm_libc::memcpy,uniform 384 to 4096
```
It is not as good as glibc for now so there's room for improvement. I suspect a path pumping 16 bytes at once given the doubled numbers for large copies.
```
BM_Memcpy/0/1 146 ns 82.5 ns 8576000 bytes_per_cycle=1.35236/s bytes_per_second=1.88922G/s items_per_second=12.1169M/s glibc memcpy,memcpy Google A
BM_Memcpy/1/1 112 ns 63.7 ns 10634240 bytes_per_cycle=0.628018/s bytes_per_second=898.387M/s items_per_second=15.702M/s glibc memcpy,memcpy Google B
BM_Memcpy/2/1 315 ns 180 ns 4079616 bytes_per_cycle=2.65229/s bytes_per_second=3.7052G/s items_per_second=5.54764M/s glibc memcpy,memcpy Google D
BM_Memcpy/3/1 85.3 ns 43.1 ns 15854592 bytes_per_cycle=0.774164/s bytes_per_second=1107.45M/s items_per_second=23.2249M/s glibc memcpy,memcpy Google L
BM_Memcpy/4/1 105 ns 54.3 ns 13427712 bytes_per_cycle=0.7793/s bytes_per_second=1114.8M/s items_per_second=18.4109M/s glibc memcpy,memcpy Google M
BM_Memcpy/5/1 77.1 ns 43.2 ns 16476160 bytes_per_cycle=0.279808/s bytes_per_second=400.269M/s items_per_second=23.1428M/s glibc memcpy,memcpy Google Q
BM_Memcpy/6/1 112 ns 62.7 ns 11236352 bytes_per_cycle=0.676078/s bytes_per_second=967.137M/s items_per_second=15.9387M/s glibc memcpy,memcpy Google S
BM_Memcpy/7/1 131 ns 65.5 ns 11751424 bytes_per_cycle=0.965616/s bytes_per_second=1.34895G/s items_per_second=15.2762M/s glibc memcpy,memcpy Google U
BM_Memcpy/8/1 104 ns 55.0 ns 12314624 bytes_per_cycle=0.583336/s bytes_per_second=834.468M/s items_per_second=18.1937M/s glibc memcpy,memcpy Google W
BM_Memcpy/9/1 932 ns 466 ns 1480704 bytes_per_cycle=3.17342/s bytes_per_second=4.43321G/s items_per_second=2.14679M/s glibc memcpy,uniform 384 to 4096
```
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D150202
Currently the opcode is only valid if it is the same between all of the
ports. This is possible to violate if the opcode is places into a memory
location and then read in a non-uniform manner by the warp / wavefront.
Moving this to a compile time constant makes it impossible to break this
invariant.
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D150115
The exact set of supported values is determined by the <errno.h>
and <signal.h> headers, which don't (yet) come from llvm-libc on
Fuchsia. The mappings of SIG* and E* codes to psignal/strsignal
and perror/strerror text used in Fuchsia libc today is the same
as for Linux.
Reviewed By: abrachet
Differential Revision: https://reviews.llvm.org/D150026
This patch uses the changed interface in D149972 to make these classes
move-only. The `Port` class could be further refined to be
construct-only in a future patch, but for now this makes it more
difficult to misuse the interface.
Depends on D149972
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D149974
This patch replaces the existing `cpp::optional` type with a newer
version that has more features. This class is heavily inspired by the
old `llvm::Optional` class. Currently the limitations of this class is
that we only handle types with trivial constructors or operators.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149972
Previously we used a single port to implement the RPC. This was
sufficient for single threaded tests but can potentially cause deadlocks
when using multiple threads. The reason for this is that GPUs make no
forward progress guarantees. Therefore one group of threads waiting on
another group of threads can spin forever because there is no guarantee
that the other threads will continue executing. The typical workaround
for this is to allocate enough memory that a sufficiently large number
of work groups can make progress. As long as this number is somewhat
close to the amount of total concurrency we can obtain reliable
execution around a shared resource.
This patch enables using multiple ports by widening the arrays to a
predetermined size and indexes into them. Empty ports are currently
obtained via a trivial linker scan. This should be imporoved in the
future for performance reasons. Portions of D148191 were applied to
achieve parallel support.
Depends on D149581
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D149598
Previously this wasn't implemented because it's effectively a no-op.
However, this should be safe to emit on sm_60 architectures. It's
important because it carries semantic importance for whether or not
something can be moved. So we should always emit this instrinsic.
Differential Revision: https://reviews.llvm.org/D149923
The GPU has a different execution model to standard `_start`
implementations. On the GPU, all threads are active at the start of a
kernel. In order to correctly intitialize and call the constructors we
want single threaded semantics. Previously, this was done using a
makeshift global barrier with atomics. However, it should be easier to
simply put the portions of the code that must be single threaded in
separate kernels and then call those with only one thread. Generally,
mixing global state between kernel launches makes optimizations more
difficult, similarly to calling a function outside of the TU, but for
testing it is better to be correct.
Depends on D149527 D148943
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D149581
The execution model of the GPU expects that groups of threads will
execute in lock-step in SIMD fashion. It's both important for
performance and correctness that we treat this as the smallest possible
granularity for an RPC operation. Thus, we map multiple threads to a
single larger buffer and ship that across the wire.
This patch makes the necessary changes to support executing the RPC on
the GPU with multiple threads. This requires some workarounds to mimic
the model when handling the protocol from the CPU. I'm not completely
happy with some of the workarounds required, but I think it should work.
Uses some of the implementation details from D148191.
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D148943
This patch updates the struct dirent to be on par with glibc (by adding
a missing d_type member) and update the readdir call to use SYS_getdents64
instead of SYS_getdents.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D147738
Carefully work around not knowing the thread mask that nvptx intrinsic
functions require.
If the warp is converged when calling try_lock, a single rpc call will handle
all lanes within it. Otherwise more than one rpc call with thread masks that
compose to the unknown one will occur.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149897
Carefully work around not knowing the thread mask that nvptx intrinsic
functions require.
If the warp is converged when calling try_lock, a single rpc call will handle
all lanes within it. Otherwise more than one rpc call with thread masks that
compose to the unknown one will occur.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149897
This test fails on sm_60 because of the atomics codegen. We test atomics
indirectly with the `rpc` so we still have coverage.
Differential Revision: https://reviews.llvm.org/D149887
A target for the test named ${fq_target_name} has been added. It depends
on ${fq_target_name}.__unit__ and ${fq_target_name}.__hermetic__ as
relevant.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149730
We use a bump pointer to implement malloc for the hermetic tests.
Currently, we bump the pointer up by any amount. This means that calling
`malloc(1)` will misalign the buffer so any following `malloc(8)`
accesses will not be aligned. This causes problems in architectures
which require alignment.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149863
Will be necessary for correct locking on volta.
API-only change to help with rebasing the rest of the stack on this.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149871
Now that a large amount of GPU tests can be run in parallel, we should
document spurious failures. It is a well-known issue that launching many
GPU applications in parallel can lead to various problems, the worst of
which being an indefinite hang.
Differential Revision: https://reviews.llvm.org/D149857
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the NVPTX target to build
and run the unit tests on the GPU. Currently this only tests the ctype
tests, but adding more should be straightforward from here on.
This ran all the ctest unit tests when run on an sm_70.
Depends on D149517 D149527
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149532
This patch adds the necessary hacks to support global constructors and
destructors. This is an incredibly hacky process caused by the primary
fact that Nvidia does not provide any binary tools and very little
linker support. We first had to emit references to these functions and
their priority in D149451. Then we dig them out of the module once it's
loaded to manually create the list that the linker should have made for
us. This patch also contains a few Nvidia specific hacks, but it passes
the test, albeit with a stack size warning from `ptxas` for the
callback. But this should be fine given the resource usage of a common
test.
This also adds a dependency on LLVM to the NVPTX loader, which hopefully doesn't
cause problems with our CUDA buildbot.
Depends on D149451
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D149527
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the AMDGPU target to build
and run the unit tests on the GPU. Currently this only tests the `ctype`
tests, but adding more should be straightforward from here on.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149517
The `-v` flag means verbose and not version.
With `clang` this flag prints the version and exits successfully.
Under `GCC` this is not a valid command line so the binary exits with
an error.
Removes the redundant Ack/Data bit manipulation.
Represents the inbox/outbox state with one bit instead of two. This will
be useful if we change to a packed representation and otherwise cuts the
runtime state space from 16 to 4.
Further simplification is possible, this patch is intentionally minimal.
- can_{send,recv}_data are now in == out
- {client,server}::try_open can be factored into Process:try_open
This implements the state machine of D148191, modulo differences in atomic
ordering and fences.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149788
Prevent operation reordering with fence instead of a comment.
The mailboxes are in shared memory and the locks structure in device memory.
If the mailboxes are read and then the lock taken, the lock says nothing
about the current or future state of those mail boxes. The relaxed atomic
fetch_or can be reordered before the relaxed atomic loads of unrelated
variables unless there is a fence preventing this.
Patches both Client::try_open and Server::try_open, one of which is missing
an optimisation and the other is missing the comment, but which otherwise
could be Process::try_open followed by buffer->opcode = opcode in Client.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149790
Decimal long doubles are not commonly used, and aren't currently
supported by the algorithm used for decimal float conversions. To avoid
giving incorrect answers, this patch adds a temporary exception to print
long doubles in hexadecimal even when decimal is requested.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D148179
This patch adds the function "socket" from the header "sys/socket". It's
a simple syscall wrapper, and I plan on adding the related functions in
a followup patch.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149622
We got rid of the rounding mode here so that the hermetic tests wouldn't
depend on the system fenv.h. But this seemed to cause some bots to
break. Getting rid of this change for now, it should be fine for the CPU
builds.
Differential Revision: https://reviews.llvm.org/D149767
We previously changed this to use `nolibc` to allow it to link in
compiler builtins for the CPU build. However, these options are unused
on the GPU and create a lot of noise. Furthermore, we want to use
`nogpulib` to prevent the linking in of the vendor libraries.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149753
This patch updates the `__support` unit tests to be built as hermetic
tests. The only tests we needed to disable were because of the use of
`realloc` which is not implemented correctly as it doesn't copy the
memory.
Depends on D149745
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149750
The FPEceptMatcher.cpp file uses system utilities and includes C++
libraries. This patch pulls it out of the main `FPTestHelpers` target so
we can exclude it from hermetic only tests.
Depends on D149705
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149745
This changes the `stdlib` tests to the new `add_libc_test` framework.
This applies to all but the exit tests.
Depends on D149691
Reviewed By: sivachandra, michaelrj
Differential Revision: https://reviews.llvm.org/D149705
We previously used a more stricter -nostdlib option which was also removing
compiler-rt/libgcc.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149683
The `add_libc_test` option allows us to enable both kinds of tests in a
single option. However, some tests cannot be made hermetic with the
current approach. Such as tests that rely on system utilities or
libraries. This patch adds two options `UNIT_TEST_ONLY` and
`HERMETIC_TEST_ONLY` to offer more fine-grained control over which
version gets built. This makes it explicit which version a test supports
and why.
Depends on D149662
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149691
This changes over the string unit tests to the new `add_libc_test`
framework. The one test that wasn't changes was the memory utils because
it overran the static buffer used for the hermetic test's bump pointer.
Depends on D149656
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149662
Currently we use `assert` as a stand-in for the typical test assertions.
This is because these functions exist outside of the base test class so
we can't use the typical assertion methods. The presence of these
asserts makes it difficult to compile these tests in a standalone
format. This patch removes all occurrences.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149656
This patch addresses some of the flags and features that are currently
missing from the hermetic test support. This mostly just fixes the
`add_libc_test` option failing to find a few dependencies or missing
arguments from the previous unit test support.
Reviewed By: lntue
Differential Revision: https://reviews.llvm.org/D149629
This patch makes the necessary changes to support calling global
constructors and destructors on the GPU. The patch in D149340 allows the
`lld` linker to create the symbols pointing us to these globals. These
should be executed by a single thread, which is more difficult on the
GPU because all threads are active. I chose to use an atomic counter to
sync every thread on the GPU. This is very slow if you use more than a
few thousand threads, but for testing purposes it should be sufficient.
Depends on D149340 D149363
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149398
The implementation of the test printing currently expects a null
terminated C-string. However, the `write_to_stderr` interface uses a
string view, which doesn't need to be null terminated. This patch
changes the printing interface to directly use `fwrite` instead rather
than relying on a null terminator.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149493
