Since befaa35212dbaac39cd76a8dc748e4df7c90a0d9, the CI stably failed for
the generic-no-wide-characters build, because in no-`wchar_t` modes, the
header for `__remove_cv_t` wasn't properly included.
This PR adds the missing include of `<__type_traits/remove_cv.h>`.
As drive-by, `<__cstddef/size_t.h>` and
`<__type_traits/is_constant_evaluated.h>`, which are included by
`<cwchar>`, are also made included by `<string>` to avoid potential
regression as we're using `size_t` and
`__libcpp_is_constant_evaluated()` in `<string>`.
There is no reason to double-wrap iterators, since we can already
achieve anything we want within `__bounded_iter`itself.
This is technically ABI breaking, but people using bounded iterators
shouldn't require ABI stability currently.
Fixes#178521
The previous `_LIBCPP_ASAN_VOLATILE_WRAPPER` approach was used to
prevent
speculative loads of string data before the short/long state was
determined. This patch replaces that mechanism with a more explicit
`__annotate_memory_barrier()` using an empty volatile assembly block.
This PR is inspired by #183457 and by downstream false positive on
`__get_long_size`. It fails same way as `__get_long_pointer` before we
have
`_LIBCPP_ASAN_VOLATILE_WRAPPER`. Barrier approach avoids
expanding `_LIBCPP_ASAN_VOLATILE_WRAPPER` for size_t, and to
in general looks more readable.
I failed to create reasonable reproducer for test, I suspect it requires
precise set of compiler flags, and libc++ site_config which will be hard
to maintain in test.
`std::__to_address(__get_pointer())` is an extremely common pattern
inside `string` and is basically equivalent to `data()`, except that
`data()` only returns a non-const pointer since C++17. This patch
introduces `__data()` to back-port returning a non-const pointer.
Address sanitizer recently got a new macro __SANITIZER_DISABLE_CONTAINER_OVERFLOW__
which is intended to disable container overflow checks in libraries (either the
standard library or user libraries that might provide such checks). This patch makes
libc++ honor that macro and, in addition, cleans up how these checks are enabled for
string (which is special) by introducing a macro just for string.
rdar://166234942
When doing amortized growth we currently have separate functions for
calculating the new capacity and allocating. However, we set the size to
the calculated capacity that way instead of the actually required size.
This makes the interface quite confusing, since the size has to be set
manually. Instead, this patch refactors the function to get the
amortized growth capacity to instead allocate and set the size
correctly.
Instead of unwrapping the arguments to string's observer functions
everywhere, simply unwrap the specific argument and then forward all of
them to a single function which does the final unwrapping of `*this`.
This simplifies the code a bit.
`__erase_external_with_move` is in the dylib, so the compiler doesn't
see the definition. Marking it `noexcept` sometimes allows clang to
remove exceptions related code, improving code size slightly.
Flags that should be used both for compiling and for linking are
provided through the %{flags} substitution. Our clang-tidy tests should
be using them, not only %{compile_flags}.
Introducing this utility makes the `__grow_by{,_and_replace}`
significantly easier to understand and allows us to migrate away from
these functions in the future.
This does a couple of things:
- code that is only useful for `shrink_to_fit` is moved into that
function
- `shrink_to_fit` is simplified a bit
- `__recommend` is renamed to better reflect what the function actually
does
- `__allocate_long_buffer` asserts that the passed capacity doesn't fit
into the SSO
The Trivial Relocation feature has been removed from the C++26 working
draft. Based on discussions in Kona, it is unlikely that the
"replaceable" type concept will come back in the C++29 time frame.
Since we don't have a use for the type trait in the library at the
moment, remove the code associated to it. If we end up needing something
like it in the future, we can always add it back.
8dae17be2991cd7f0d7fd9aa5aecd064520a14f6 refactors basic_string for more
code reuse. This makes sense in most cases, but has performance overhead
in the case of ~basic_string. The refactoring of ~basic_string to call
__reset_internal_buffer() added a redundant (inside the destructor)
reset of the object, which the optimizer is unable to optimize away in
many cases. This patch prevents a ~1% regression we observed on an
internal workload when applying the original refactoring. This does
slightly pessimize the code readability, but I think this change is
worth it given the performance impact.
I'm hoping to add a benchmark(s) to the upstream libc++ benchmark suite
around string construction/destruction to ensure that this case does not
regress as it seems common in real world applications. I will put up a
separate PR for that when I figure out a reasonable way to write it.
Verify that the operation passed to resize_and_overwrite returns an
integer-like type, matching the behavior of other standard library
implementations like GCC's libstdc++
Fixes#160577
There is a tradition to use U.S. English spellings for APIs. For
example, it's uninitialized_fill and not uninitialised_fill,
specialization not specialisation, etcetera.
This patch adds additional overloads to `map::at` in case its known that
the argument is transparently comparable to the key type. This avoids
actually constructing the key type in some cases, potentially removing
allocations.
```
--------------------------------------------------------
Benchmark old new
--------------------------------------------------------
BM_map_find_string_literal 12.8 ns 2.68 ns
```
This patch introduces a string-internal API to make the allocation and
deallocation the long string simpler. Before this we had a lot of code
duplication, so ensuring that things were actually correct was
non-trivial.
Since `__resize_default_init` is only ever used inside the dylib we can
remove the libc++-internal API and switch to the public one. This patch
inlines a bunch of functions that aren't required anymore and simplifies
the code that way.
This uses the new `__builtin_lt_synthesises_from_spaceship` builtin from
clang to use three way comparison for arbitrary user-defined types that
only provide a spaceship operator.
We've built up quite a few links directly to github within the code
base. We should instead use `llvm.org/PR<issue-number>` to link to bugs,
since that is resilient to the bug tracker changing in the future. This
is especially relevant for tests linking to bugs, since they will
probably be there for decades to come. A nice side effect is that these
links are significantly shorter than the GH links, making them much less
of an eyesore.
This patch also replaces a few links that linked to the old bugzilla
instance on llvm.org.
LLVM 20 contained an ABI break that can result in the size of
`std::unordered_{map,set,multimap,multiset}` and `std::deque` changing
when used with an allocator type that is empty and contains a base class
that is the same across rebound allocator instantiations (e.g.
``Allocator<int>`` and ``Allocator<char>`` are both empty and contain
the same base class).
In addition, the layout of a user-defined type that:
- contains one of the following containers:
`std::unordered_{map,set,multimap,multiset}`, `std::deque`, `std::map`,
`std::set`, `std::multimap`, `std::multiset`, `std::list` or
`std::vector`, and
- passes an empty allocator, comparator or hasher type to that
container, and
- has a member of that same empty allocator, comparator or hasher type
inside the enclosing struct, and
- that member is either marked with `[[no_unique_address]]` or optimized
out via the EBO (empty base optimization) technique
saw its size increase from LLVM 19 to LLVM 20. This was caused by the
usage of `[[no_unique_address]]` within some of libc++'s containers in a
way that allowed subtle interactions with enclosing objects. This is
fixed in LLVM 21 on Clang (returning to the LLVM 19 ABI), however that
implies an ABI break from LLVM 20 to LLVM 21.
Furthermore, fixing this causes a slight regression to constant
evaluation support in `std::unique_ptr`. Specifically, constant
evaluation will now fail when the deleter relies on being
value-initialized for constant-evaluation admissibility. If a
default-initialized deleter can be used during constant evaluation, or
if the default constructor is non-trivial, the `unique_ptr` is not
affected by this regression. In particular, this regression does not
impact any `unique_ptr` using the default deleter.
Note that there is currently no way to realistically fix this ABI break
on GCC, therefore GCC will remain on the ABI introduced in LLVM 19. That
also means that Clang and GCC will have a slightly different ABI for the
small subset of types listed above until we are able to apply the same
fix we did with Clang on GCC.
We fix this regression by surrounding the members of the
`_LIBCPP_COMPRESSED_PAIR` with an anonymous struct. This restricts the
shifting of empty types to the front of the `_LIBCPP_COMPRESSED_PAIR`
instead of throughout the surrounding object. This "frees up" the zero
offset to contain another object of the same type, restoring the ability
to perform EBO or to elide the storage for a type with
`[[no_unique_address]]` in the enclosing (user-defined) struct.
Fixes#154146
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
The use-case for `__is_same_uncvref` seems rather dubious, since not a
single use-cases needed the `remove_cvref_t` to be applied to both of
the arguments. Removing the alias makes it clearer what actually
happens, since we're not using an internal name anymore and it's clear
what the `remove_cvref_t` should apply to.
Currently, when the string shrink into the SSO buffer, the `__rep_.__s`
member isn't activated before the `traits_type::copy` call
yet, so internal `__builtin_memmove` call writing to the buffer causes
constant evaluation failure. The existing test coverage seems a bit
defective and doesn't cover this case - `shrink_to_fit` is called on the
copy of string after erasure, not the original string object.
This PR reorders the `__set_short_size` call, which starts the lifetime
of the SSO buffer, before the copy operation. Test coverage is achieved
by calling `shrink_to_fit` on the original erased string.
This reverts commit c861fe8a71e64f3d2108c58147e7375cd9314521.
Unfortunately, this use of hidden visibility attributes causes
user-defined specializations of standard-library types to also be marked
hidden by default, which is incorrect. See discussion thread on #131156.
...and also reverts the follow-up commits:
Revert "[libc++] Add explicit ABI annotations to functions from the block runtime declared in <__functional/function.h> (#140592)"
This reverts commit 3e4c9dc299c35155934688184319d391b298fff7.
Revert "[libc++] Make ABI annotations explicit for windows-specific code (#140507)"
This reverts commit f73287e623a6c2e4a3485832bc3e10860cd26eb5.
Revert "[libc++][NFC] Replace a few "namespace std" with the correct macro (#140510)"
This reverts commit 1d411f27c769a32cb22ce50b9dc4421e34fd40dd.
This patch introduces `_LIBCPP_{BEGIN,END}_EXPLICIT_ABI_ANNOTATIONS`,
which allow us to have implicit annotations for most functions, and just
where it's not "hide_from_abi everything" we add explicit annotations.
This allows us to drop the `_LIBCPP_HIDE_FROM_ABI` macro from most
functions in libc++.
That type trait represents whether move-assigning an object is
equivalent to destroying it and then move-constructing a new one from
the same argument. This will be useful in a few places where we may want
to destroy + construct instead of doing an assignment, in particular
when implementing some container operations in terms of relocation.
This is effectively adding a library emulation of P2786R12's
is_replaceable trait, similarly to what we do for trivial relocation.
Eventually, we can replace this library emulation by the real
compiler-backed trait.
This is building towards #129328.
There are currently lots of `_LIBCPP_HAS_ASAN` and
`__libcpp_is_constant_evaluated()` checks which aren't needed, since it
is centrally checked inside `__debug_utils/sanitizers.h`.
This patch moves the functions common between our two extern template
lists into a common list and avoids some unnecessary _Uglification. This
makes the code a lot nicer to read and makes the differences between the
two lists obvious.
Previously, the segmented iterator optimization for `std::for_each` was restricted to C++23 and later due to its dependency on `__movable_box`, which is not available in earlier standards. This patch eliminates that restriction, enabling consistent optimizations starting from C++11.
By backporting this enhancement, we improve performance across older standards and create opportunities to extend similar optimizations to other algorithms by forwarding their calls to `std::for_each`.
Requirements on character-like types are updated unconditionally,
because `basic_string` does requires the default-constructibility. It
might be possible to make `basic_string_view` support classes with
non-public trivial default constructor, but this doesn't seem sensible.
libcxxabi's `ItaniumDemangle.h` is also updated to avoid deprecated
features.
This macro isn't required if we define all the functions inline. In
fact, quite a few of the marked functions have already been inlined.
This patch basically only moves code around and adds
`_LIBCPP_HIDE_FROM_ABI` to the places where it's been missing so far.
This also removes inlining hints, since it dropps `inline` in some
places, but that shouldn't make much of a difference. The functions tend
to be either really small, so should be inlined anyways, or are big
enough that they shouldn't be inlined even with an inlinehint.
This PR refactors `basic_string` a bit to simplify its implementation in
the following ways:
- Instead of manually checking whether a string is short or long,
followed by calling the specific functions (e.g., `__get_short_size()`,
`__get_long_size()`), we call the general functions (`size()`) to hide
the conditional checks and make the code more concise.
- Once a string is determined to be short or long, we directly call the
specific functions instead of the general versions to get rid of
unnecessary internal conditional checks. For example, for a long string,
we would directly call `{__set, __get}_long_pointer` instead of `{__set,
__get}_pointer()`.
- Variables that are defined in both the `if` and `else` branches are
now declared in a common scope to reduce redundancy.
- When the string size is calculated multiple times using
`traits_type::length(__s)`, a variable is introduced to store its
length. While modern compilers can optimize this with constant folding,
explicitly storing the length improves code readability and makes the
logic clearer.
- Fixed synopsis with missing default arguments.
`std::erase(_if)` for `basic_string` were made `constexpr` in C++20 by
cplusplus/draft@2c1ab9775c as follow-up
changes of P0980R1.
This patch implements the missed changes that were not tracked in a
specific paper.
