This patch adds two versions of memcpy optimized for architectures where unaligned accesses are either illegal or extremely slow.
It is currently enabled for RISCV 64 and RISCV 32 but it could be used for ARM 32 architectures as well.
Here is the before / after output of `libc.benchmarks.memory_functions.opt_host --benchmark_filter=BM_Memcpy` on a quad core Linux starfive RISCV 64 board running at 1.5GHz.
Before:
```
Run on (4 X 1500 MHz CPU s)
CPU Caches:
L1 Instruction 32 KiB (x4)
L1 Data 32 KiB (x4)
L2 Unified 2048 KiB (x1)
------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------
BM_Memcpy/0/0 474 ns 474 ns 1483776 bytes_per_cycle=0.243492/s bytes_per_second=348.318M/s items_per_second=2.11097M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0 210 ns 209 ns 3649536 bytes_per_cycle=0.233819/s bytes_per_second=334.481M/s items_per_second=4.77519M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0 1814 ns 1814 ns 396288 bytes_per_cycle=0.247899/s bytes_per_second=354.622M/s items_per_second=551.402k/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0 89.3 ns 89.2 ns 7459840 bytes_per_cycle=0.217415/s bytes_per_second=311.014M/s items_per_second=11.2071M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0 134 ns 134 ns 3815424 bytes_per_cycle=0.226584/s bytes_per_second=324.131M/s items_per_second=7.44567M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0 52.8 ns 52.6 ns 11001856 bytes_per_cycle=0.194893/s bytes_per_second=278.797M/s items_per_second=19.0284M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0 180 ns 180 ns 4101120 bytes_per_cycle=0.231884/s bytes_per_second=331.713M/s items_per_second=5.55957M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0 195 ns 195 ns 3906560 bytes_per_cycle=0.232951/s bytes_per_second=333.239M/s items_per_second=5.1217M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0 152 ns 152 ns 4789248 bytes_per_cycle=0.227507/s bytes_per_second=325.452M/s items_per_second=6.58187M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0 6036 ns 6033 ns 118784 bytes_per_cycle=0.249158/s bytes_per_second=356.423M/s items_per_second=165.75k/s __llvm_libc::memcpy,uniform 384 to 4096
```
After:
```
BM_Memcpy/0/0 126 ns 126 ns 5770240 bytes_per_cycle=1.04707/s bytes_per_second=1.46273G/s items_per_second=7.9385M/s __llvm_libc::memcpy,memcpy Google A
BM_Memcpy/1/0 75.1 ns 75.0 ns 10204160 bytes_per_cycle=0.691143/s bytes_per_second=988.687M/s items_per_second=13.3289M/s __llvm_libc::memcpy,memcpy Google B
BM_Memcpy/2/0 333 ns 333 ns 2174976 bytes_per_cycle=1.39297/s bytes_per_second=1.94596G/s items_per_second=3.00002M/s __llvm_libc::memcpy,memcpy Google D
BM_Memcpy/3/0 49.6 ns 49.5 ns 16092160 bytes_per_cycle=0.710161/s bytes_per_second=1015.89M/s items_per_second=20.1844M/s __llvm_libc::memcpy,memcpy Google L
BM_Memcpy/4/0 57.7 ns 57.7 ns 11213824 bytes_per_cycle=0.561557/s bytes_per_second=803.314M/s items_per_second=17.3228M/s __llvm_libc::memcpy,memcpy Google M
BM_Memcpy/5/0 48.0 ns 47.9 ns 16437248 bytes_per_cycle=0.346708/s bytes_per_second=495.97M/s items_per_second=20.8571M/s __llvm_libc::memcpy,memcpy Google Q
BM_Memcpy/6/0 67.5 ns 67.5 ns 10616832 bytes_per_cycle=0.614173/s bytes_per_second=878.582M/s items_per_second=14.8142M/s __llvm_libc::memcpy,memcpy Google S
BM_Memcpy/7/0 84.7 ns 84.6 ns 10480640 bytes_per_cycle=0.819077/s bytes_per_second=1.14424G/s items_per_second=11.8174M/s __llvm_libc::memcpy,memcpy Google U
BM_Memcpy/8/0 61.7 ns 61.6 ns 11191296 bytes_per_cycle=0.550078/s bytes_per_second=786.893M/s items_per_second=16.2279M/s __llvm_libc::memcpy,memcpy Google W
BM_Memcpy/9/0 981 ns 981 ns 703488 bytes_per_cycle=1.52333/s bytes_per_second=2.12807G/s items_per_second=1019.81k/s __llvm_libc::memcpy,uniform 384 to 4096
```
It is not as good as glibc for now so there's room for improvement. I suspect a path pumping 16 bytes at once given the doubled numbers for large copies.
```
BM_Memcpy/0/1 146 ns 82.5 ns 8576000 bytes_per_cycle=1.35236/s bytes_per_second=1.88922G/s items_per_second=12.1169M/s glibc memcpy,memcpy Google A
BM_Memcpy/1/1 112 ns 63.7 ns 10634240 bytes_per_cycle=0.628018/s bytes_per_second=898.387M/s items_per_second=15.702M/s glibc memcpy,memcpy Google B
BM_Memcpy/2/1 315 ns 180 ns 4079616 bytes_per_cycle=2.65229/s bytes_per_second=3.7052G/s items_per_second=5.54764M/s glibc memcpy,memcpy Google D
BM_Memcpy/3/1 85.3 ns 43.1 ns 15854592 bytes_per_cycle=0.774164/s bytes_per_second=1107.45M/s items_per_second=23.2249M/s glibc memcpy,memcpy Google L
BM_Memcpy/4/1 105 ns 54.3 ns 13427712 bytes_per_cycle=0.7793/s bytes_per_second=1114.8M/s items_per_second=18.4109M/s glibc memcpy,memcpy Google M
BM_Memcpy/5/1 77.1 ns 43.2 ns 16476160 bytes_per_cycle=0.279808/s bytes_per_second=400.269M/s items_per_second=23.1428M/s glibc memcpy,memcpy Google Q
BM_Memcpy/6/1 112 ns 62.7 ns 11236352 bytes_per_cycle=0.676078/s bytes_per_second=967.137M/s items_per_second=15.9387M/s glibc memcpy,memcpy Google S
BM_Memcpy/7/1 131 ns 65.5 ns 11751424 bytes_per_cycle=0.965616/s bytes_per_second=1.34895G/s items_per_second=15.2762M/s glibc memcpy,memcpy Google U
BM_Memcpy/8/1 104 ns 55.0 ns 12314624 bytes_per_cycle=0.583336/s bytes_per_second=834.468M/s items_per_second=18.1937M/s glibc memcpy,memcpy Google W
BM_Memcpy/9/1 932 ns 466 ns 1480704 bytes_per_cycle=3.17342/s bytes_per_second=4.43321G/s items_per_second=2.14679M/s glibc memcpy,uniform 384 to 4096
```
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D150202
Currently the opcode is only valid if it is the same between all of the
ports. This is possible to violate if the opcode is places into a memory
location and then read in a non-uniform manner by the warp / wavefront.
Moving this to a compile time constant makes it impossible to break this
invariant.
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D150115
The execution model of the GPU expects that groups of threads will
execute in lock-step in SIMD fashion. It's both important for
performance and correctness that we treat this as the smallest possible
granularity for an RPC operation. Thus, we map multiple threads to a
single larger buffer and ship that across the wire.
This patch makes the necessary changes to support executing the RPC on
the GPU with multiple threads. This requires some workarounds to mimic
the model when handling the protocol from the CPU. I'm not completely
happy with some of the workarounds required, but I think it should work.
Uses some of the implementation details from D148191.
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D148943
This test fails on sm_60 because of the atomics codegen. We test atomics
indirectly with the `rpc` so we still have coverage.
Differential Revision: https://reviews.llvm.org/D149887
We use a bump pointer to implement malloc for the hermetic tests.
Currently, we bump the pointer up by any amount. This means that calling
`malloc(1)` will misalign the buffer so any following `malloc(8)`
accesses will not be aligned. This causes problems in architectures
which require alignment.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149863
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the NVPTX target to build
and run the unit tests on the GPU. Currently this only tests the ctype
tests, but adding more should be straightforward from here on.
This ran all the ctest unit tests when run on an sm_70.
Depends on D149517 D149527
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149532
This patch adds the necessary hacks to support global constructors and
destructors. This is an incredibly hacky process caused by the primary
fact that Nvidia does not provide any binary tools and very little
linker support. We first had to emit references to these functions and
their priority in D149451. Then we dig them out of the module once it's
loaded to manually create the list that the linker should have made for
us. This patch also contains a few Nvidia specific hacks, but it passes
the test, albeit with a stack size warning from `ptxas` for the
callback. But this should be fine given the resource usage of a common
test.
This also adds a dependency on LLVM to the NVPTX loader, which hopefully doesn't
cause problems with our CUDA buildbot.
Depends on D149451
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D149527
The previous patches added the necessary support for global constructors
used to register tests. This patch enables the AMDGPU target to build
and run the unit tests on the GPU. Currently this only tests the `ctype`
tests, but adding more should be straightforward from here on.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149517
Removes the redundant Ack/Data bit manipulation.
Represents the inbox/outbox state with one bit instead of two. This will
be useful if we change to a packed representation and otherwise cuts the
runtime state space from 16 to 4.
Further simplification is possible, this patch is intentionally minimal.
- can_{send,recv}_data are now in == out
- {client,server}::try_open can be factored into Process:try_open
This implements the state machine of D148191, modulo differences in atomic
ordering and fences.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149788
This patch adds the function "socket" from the header "sys/socket". It's
a simple syscall wrapper, and I plan on adding the related functions in
a followup patch.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149622
We got rid of the rounding mode here so that the hermetic tests wouldn't
depend on the system fenv.h. But this seemed to cause some bots to
break. Getting rid of this change for now, it should be fine for the CPU
builds.
Differential Revision: https://reviews.llvm.org/D149767
This patch updates the `__support` unit tests to be built as hermetic
tests. The only tests we needed to disable were because of the use of
`realloc` which is not implemented correctly as it doesn't copy the
memory.
Depends on D149745
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149750
The FPEceptMatcher.cpp file uses system utilities and includes C++
libraries. This patch pulls it out of the main `FPTestHelpers` target so
we can exclude it from hermetic only tests.
Depends on D149705
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149745
This changes the `stdlib` tests to the new `add_libc_test` framework.
This applies to all but the exit tests.
Depends on D149691
Reviewed By: sivachandra, michaelrj
Differential Revision: https://reviews.llvm.org/D149705
The `add_libc_test` option allows us to enable both kinds of tests in a
single option. However, some tests cannot be made hermetic with the
current approach. Such as tests that rely on system utilities or
libraries. This patch adds two options `UNIT_TEST_ONLY` and
`HERMETIC_TEST_ONLY` to offer more fine-grained control over which
version gets built. This makes it explicit which version a test supports
and why.
Depends on D149662
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149691
This changes over the string unit tests to the new `add_libc_test`
framework. The one test that wasn't changes was the memory utils because
it overran the static buffer used for the hermetic test's bump pointer.
Depends on D149656
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149662
Currently we use `assert` as a stand-in for the typical test assertions.
This is because these functions exist outside of the base test class so
we can't use the typical assertion methods. The presence of these
asserts makes it difficult to compile these tests in a standalone
format. This patch removes all occurrences.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149656
This patch addresses some of the flags and features that are currently
missing from the hermetic test support. This mostly just fixes the
`add_libc_test` option failing to find a few dependencies or missing
arguments from the previous unit test support.
Reviewed By: lntue
Differential Revision: https://reviews.llvm.org/D149629
This patch makes the necessary changes to support calling global
constructors and destructors on the GPU. The patch in D149340 allows the
`lld` linker to create the symbols pointing us to these globals. These
should be executed by a single thread, which is more difficult on the
GPU because all threads are active. I chose to use an atomic counter to
sync every thread on the GPU. This is very slow if you use more than a
few thousand threads, but for testing purposes it should be sufficient.
Depends on D149340 D149363
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149398
The time functions have not yet been updated to match our new coding
patterns. This patch removes some unnecessary includes, adjusts the
names of the test targets, and adds several TODO comments. It is my
intention to follow this patch up with additional cleanup.
Reviewed By: sivachandra, rtenneti
Differential Revision: https://reviews.llvm.org/D149487
The existing LibcTestMain has been renamed to LibcUnitTestMain.
Hermetic tests are linked to LibcHermeticTestMain and unit tests are
linked to LibcUnitTestMain.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D149303
Summary:
This patch makes this only apply to the GPU build. This should be
handled more intelligently in the future so it's common between all of t
hem.
Summary:
This is a little broken, what we really need is a separate target to use
with the hermetic tests, but this is a stop-gap to get the bots green
again.
The previous patch in D149216 allows us to use the internal `<stdlib.h>`
include for the GPU build. However, we currently don't provide the
memory functions so the header wasn't resolving them. This patch adds
these as entrypoints. They don't cause any entrypoints to be emitted
because they are not implemented, but they provide it in the header so
that we can rely on the test's implementation of them.
Depends on D149216
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149217
The generated header files live in the build directory's include path.
When targeting a hermetic build we want to make sure we only use headers
generated by the project itself if availible.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D149216
This patch adds targets for printf and fprintf to the bazel build.
Additionally, it adds support for the build system to specify where
files should be written for testing purposes. This was necessary to
enable the fprintf test under bazel.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D147008
Previously used SYS_sched_get_priority_min. This also updates the tests
for SCHED_RR and SCHED_FIFO to check max_priority > min_priority, rather
than >= in order to catch future breakages.
Reviewed By: michaelrj, goldstein.w.n
Differential Revision: https://reviews.llvm.org/D149102
A convenience wrapper name `add_libc_test` is also added which adds both
a unit test and a hermetic test. The ctype tests have been switched over
to use add_libc_test.
Reviewed By: jhuber6
Differential Revision: https://reviews.llvm.org/D148756
Previously, only those unit tests which belonged to a suite were run as
part of libc-unit-tests. It meant that unit tests not part of any suite
were not being tested. This change makes all unit tests run as part of
libc-unit-tests. The convenience function to add a libc unit test suite
has been removed and add_custom_target is used in its place. One of the
bit-rotting test has along been fixed. Math exhaustive and differential
tests are skipped under full build.
Reviewed By: michaelrj
Differential Revision: https://reviews.llvm.org/D148784
This patch adds extra intrinsics for the GPU. Some of these are unused
for now but will be used later. We use these currently to update the
`RPC` handling. Currently, every thread can update the RPC client, which
isn't correct. This patch adds code neccesary to allow a single thread
to perfrom the write while the others wait.
Feedback is welcome for the naming of these functions. I'm copying the
OpenMP nomenclature where we call an AMD `wavefront` or NVIDIA `warp` a
`lane`.
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D148810
Uint addition and subtraction normally use builtins which aren't
constexpr. This patch adds an rvalue overload version of the addition
and subtraction operation that is always constexpr.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D148759
Previously unconditionally stored to the return value. This is
incorrect, we should only return if user value is non-null.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D148293
The RPC interface can support multiple independent clients. This support
currently only supports many single-thread warps / workgroups
coordinating over a single lock. This patch uses the support added in
the previous patch to test the RPC interface with multiple blocks.
Note that this does not work with multiple threads currently because of
the effect of warps / workgroups executing in lockstep incorrectly. This
will be added later.
Depends on D148485
Reviewed By: lntue, sivachandra
Differential Revision: https://reviews.llvm.org/D148486
Currently, the RPC interface with the loader is only tested if the other
tests fail. This test adds a direct test that runs a simple integer
increment over the RPC handshake 10000 times.
Depends on https://reviews.llvm.org/D148288
Reviewed By: lntue
Differential Revision: https://reviews.llvm.org/D148342
Same issue as was fixed in commit 3d95323, but for hexadecimal floats.
Reviewed By: sivachandra
Differential Revision: https://reviews.llvm.org/D148545
