`-Wchanges-meaning` is a GCC warning that catches shadowing in more
contexts. While a bit annoying here, it's a helpful warning. As such, we
need to rename the `__split_buffer` alias template in `std::vector` so
that we don't trip it up.
`__add_alignment_assumption` unilaterally returned a `pointer`, even
when passed a `const_pointer`. This was surfaced by some (but not all)
CI jobs when rebasing `std::vector` to have a layout type.
Currently, `deque` and `vector`'s `append_range` is implemented in terms
of `insert_range`. The problem with that is that `insert_range` has more
preconditions, resulting in us rejecting valid code.
This also significantly improves performance for `deque` in some cases.
This both simplifies the implementation and improves the performance,
since the compiler is better able to see through what's going on.
```
Benchmark old new Difference % Difference
-------------------------------------------------------------------------- -------------- -------------- ------------ --------------
vector<bool>(const_vector<bool>&) 11.99 12.26 0.27 2.25%
vector<bool>(size_type,_const_value_type&) 9.24 9.29 0.05 0.54%
vector<bool>(vector<bool>&&,_const_allocator_type&)_(different_allocators) 14.26 14.35 0.09 0.65%
vector<bool>(vector<bool>&&,_const_allocator_type&)_(equal_allocators) 2.67 2.67 -0.01 -0.29%
vector<bool>::reserve() 9.30 9.29 -0.01 -0.12%
vector<bool>::resize() 15.14 13.43 -1.71 -11.28%
Geomean 9.17 9.03 -0.14 -1.48%
Allocators should be extremely cheap, if not free, to copy. Furthermore,
we have requirements on allocator types that copies must compare equal,
and that move and copy must be the same.
Hence, taking an allocator by reference should not provide benefits
beyond making a copy of it. However, taking the allocator by reference
leads to complexity in __split_buffer, which can be removed if we stop
using that pattern.
This reduces the amount of code we have to maintain a bit.
This also simplifies `vector` by using the internal API instead of
`#if`s to switch based on language dialect.
The Trivial Relocation feature has been removed from the C++26 working
draft. Based on discussions in Kona, it is unlikely that the
"replaceable" type concept will come back in the C++29 time frame.
Since we don't have a use for the type trait in the library at the
moment, remove the code associated to it. If we end up needing something
like it in the future, we can always add it back.
#119632 introduced a code size and performance regression. This was
verified by running the included benchmark against trunk and trunk with
#119632 reverted. Instead of actually reverting that patch, we can
inline `__construct_at_end` into the few places it's used instead,
simplifying the implementation further (by not handling special cases we
never actually encounter).
```
Benchmark Baseline Candidate Difference % Difference
------------------------ ---------- ----------- ------------ --------------
BM_vector_bool_size_ctor 29.91 8.56 -21.35 -71.37
```
This reverts commit 1bafd020c7c80be476f211bc239ce43424f7e0ce.
This breaks the LLDB data formatters which means these failures show up
on every premerge run. Reverting for now until fixing the LLDB
formatters can be coordinated with a relanding.
**tl;dr** We can significantly improve the runtime performance of
`std::vector` by changing its representation from three pointers to one
pointer and two integers. This document explains the details of this
change, along with the justifications for making it. See the [RFC] for
more information.
`vector` depends on `__split_buffer` for inserting elements. Changing
`__split_buffer` to match `vector`'s representation simplifies the
model, as it eliminates the need to convert between two different
representations of a contiguous buffer in the same configuration of
libc++.
[RFC]: https://discourse.llvm.org/t/adding-a-size-based-vector-to-libc-s-unstable-abi/86306
---------
Co-authored-by: Jorge Gorbe Moya <jgorbe@google.com>
These are almost certainly intended to not be inlined. This
significantly reduces code size when `push_back` and `emplace_back` are
used heavily.
Fixes#94360
We've built up quite a few links directly to github within the code
base. We should instead use `llvm.org/PR<issue-number>` to link to bugs,
since that is resilient to the bug tracker changing in the future. This
is especially relevant for tests linking to bugs, since they will
probably be there for decades to come. A nice side effect is that these
links are significantly shorter than the GH links, making them much less
of an eyesore.
This patch also replaces a few links that linked to the old bugzilla
instance on llvm.org.
This is brought up in the LWG reflector. We currently call `reserve` if
the underlying container has one. But the spec does not specify what
`reserve` should do for Sequence Container. So in theory if the
underlying container is user defined type and it can have a function
called `reserve` which does something completely different.
The fix is to just call `reserve` for STL containers if it has one
libc++ makes the `hash<vector<bool, A>>::operator()` `constexpr` since
C++20, which is a conforming extension, but it was unintended.
This patch removes the extension, with an escape hatch macro for it, and
the escape hatch will be removed in the future. Test cases for
`constexpr` along with the assumption of hash values are moved to the
`libcxx/test/libcxx/` subdirectory.
---------
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
That type trait represents whether move-assigning an object is
equivalent to destroying it and then move-constructing a new one from
the same argument. This will be useful in a few places where we may want
to destroy + construct instead of doing an assignment, in particular
when implementing some container operations in terms of relocation.
This is effectively adding a library emulation of P2786R12's
is_replaceable trait, similarly to what we do for trivial relocation.
Eventually, we can replace this library emulation by the real
compiler-backed trait.
This is building towards #129328.
There are currently lots of `_LIBCPP_HAS_ASAN` and
`__libcpp_is_constant_evaluated()` checks which aren't needed, since it
is centrally checked inside `__debug_utils/sanitizers.h`.
The current implementation of `{std, ranges}::equal` fails to correctly
compare `vector<bool>`s when the underlying storage type is smaller than
`int` (e.g., `unsigned char`, `unsigned short`, `uint8_t` and
`uint16_t`). See [demo](https://godbolt.org/z/j4s87s6b3)). The problem
arises due to integral promotions on the intermediate bitwise
operations, leading to incorrect final equality comparison results. This
patch fixes the issue by ensuring that `{std, ranges}::equal` operate
properly for both aligned and unaligned bits.
Fixes#126369.
This PR optimizes the performance of `std::ranges::rotate` for
`vector<bool>::iterator`. The optimization yields a performance
improvement of up to 2096x.
Closes#64038.
This PR optimizes the performance of `std::ranges::equal` for
`vector<bool>::iterator`, addressing a subtask outlined in issue #64038.
The optimizations yield performance improvements of up to 188x for
aligned equality comparison and 82x for unaligned equality
comparison. Moreover, comprehensive tests covering up to 4 storage words
(256 bytes) with odd and even bit sizes are provided, which validate the
proposed optimizations in this patch.
This is technically not necessary in most cases to prevent issues with ADL,
but let's be consistent. This allows us to remove the libcpp-qualify-declval
clang-tidy check, which is now enforced by the robust-against-adl clang-tidy check.
This is to fix compile error with explicit Clang modules like
```
../../third_party/libc++/src/include/__vector/vector_bool.h:85:11: error: default argument of '__bit_iterator' must be imported from module 'std.bit_reference_fwd' before it is required
85 | typedef __bit_iterator<vector, false> pointer;
| ^
../../third_party/libc++/src/include/__fwd/bit_reference.h:23:68: note: default argument declared here is not reachable
23 | template <class _Cp, bool _IsConst, typename _Cp::__storage_type = 0>
| ^
```
This PR slightly simplifies the implementation of `vector<bool>::max_size`
and adds extensive tests for the `max_size()` function for both `vector<bool>`
and `vector<T>`. The main purposes of the new tests include:
- Verify correctness of `max_size()` under various `size_type` and
`difference_type` definitions: check that `max_size()` works properly
with allocators that have custom `size_type` and `difference_type`. This
is particularly useful for `vector<bool>`, as different `size_type` lead
to different `__storage_type` of different word lengths, resulting in
varying `max_size()` values for `vector<bool>`. Additionally, different
`difference_type` also sets different upper limit of `max_size()` for
both `vector<bool>` and `std::vector`. These tests were previously
missing.
- Eliminate incorrect implementations: Special tests are added to identify and
reject incorrect implementations of `vector<bool>::max_size` that unconditionally
return `std::min<size_type>(size-max, __internal_cap_to_external(allocator-max-size))`.
This can cause overflow in the `__internal_cap_to_external()` call and lead
to incorrect results. The new tests ensure that such incorrect
implementations are identified.
There's no reason not to, and it's easy enough to do using enable_if. As
a drive-by change, also add a missing _LIBCPP_NO_CFI attribute on
__add_alignment_assumption.
As a follow-up to #121013 (which focused on `std::ranges::copy`), this
PR optimizes the performance of `std::ranges::copy_backward` for
`vector<bool>::iterator`, addressing a subtask outlined in issue #64038.
The optimizations yield performance improvements of up to 2000x for
aligned copies and 60x for unaligned copies.
This patch simplifies the implementation of `__construct_at_end` in
`vector<bool>`, which currently contains duplicate initialization logic
across its two overloads.
Changes:
- Carve out sized but input-only ranges for C++23.
- Call `std::move` for related functions when the iterator is possibly input-only.
Fixes#115727
This PR addresses an issue where the `shrink_to_fit` function in
`vector<bool>` is effectively a no-op, meaning it will never shrink the
capacity.
Fixes#122502
In https://reviews.llvm.org/D136765 / https://reviews.llvm.org/D144155,
the asan annotations for `std::vector` were modified to unpoison freed
backing memory on destruction, instead of leaving it poisoned. However,
calling `__clear()` instead of `clear()` skips informing the asan runtime
of this decrease in the accessible container size, which breaks the
invariant that the value of `old_mid` should match the value of `new_mid`
from the previous call to `__sanitizer_annotate_contiguous_container`, which
can trip the sanity checks for the partial poison between [d1, d2) and the
container redzone between [d2, c), if enabled. To fix this, ensure that
`clear()` is called instead, as is already done by `__vdeallocate()`.
Also remove `__clear()`, since it is no longer called.
Missing information about begin and end pointers of std::vector can lead
to missed optimizations in LLVM.
This patch adds alignment assumptions at the point where the begin and
end pointers are loaded. If the pointers would not have the same
alignment, end might never get hit when incrementing begin.
See https://github.com/llvm/llvm-project/issues/101372 for a discussion
of missed range check optimizations in hardened mode.
Once https://github.com/llvm/llvm-project/pull/108958 lands, the created
`llvm.assume` calls for the alignment should be folded into the `load`
instructions, resulting in no extra instructions after InstCombine.
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
As a follow-up to #113852, this PR optimizes the performance of the
`insert(const_iterator pos, InputIt first, InputIt last)` function for
`input_iterator`-pair inputs in `std::vector` for cases where
reallocation occurs during insertion. Additionally, this optimization
enhances exception safety by replacing the traditional `try-catch`
mechanism with a modern exception guard for the `insert` function.
The optimization targets cases where insertion trigger reallocation. In
scenarios without reallocation, the implementation remains unchanged.
Previous implementation
-----------------------
The previous implementation of `insert` is inefficient in reallocation
scenarios because it performs the following steps separately:
- `reserve()`: This leads to the first round of relocating old
elements to new memory;
- `rotate()`: This leads to the second round of reorganizing the
existing elements;
- Move-forward: Moves the elements after the insertion position to
their final positions.
- Insert: performs the actual insertion.
This approach results in a lot of redundant operations, requiring the
elements to undergo three rounds of relocations/reorganizations to be
placed in their final positions.
Proposed implementation
-----------------------
The proposed implementation jointly optimize the above 4 steps in the
previous implementation such that each element is placed in its final
position in just one round of relocation. Specifically, this
optimization reduces the total cost from 2 relocations + 1 std::rotate
call to just 1 relocation, without needing to call `std::rotate`,
thereby significantly improving overall performance.
