Currently, when the string shrink into the SSO buffer, the `__rep_.__s`
member isn't activated before the `traits_type::copy` call
yet, so internal `__builtin_memmove` call writing to the buffer causes
constant evaluation failure. The existing test coverage seems a bit
defective and doesn't cover this case - `shrink_to_fit` is called on the
copy of string after erasure, not the original string object.
This PR reorders the `__set_short_size` call, which starts the lifetime
of the SSO buffer, before the copy operation. Test coverage is achieved
by calling `shrink_to_fit` on the original erased string.
In 0547e573c555, I introduced backdeployment testing on macOS using
Github-provided builders. This was done by basically building libc++ on
a slightly older macOS (like macOS 13) and then running against the
system library on that machine. However, that created a dependency that
libc++ must keep working on macOS 13, which doesn't support the
latest-released Xcode.
This patch solves that problem by moving the deployment testing to a
newer version of macOS which supports the latest-released version of
Xcode.
Sadly, that also reduces the backdeployment coverage we have since we're
not actually testing on older OSes, but is necessary to satisfy the
documented libc++ support policy. In the future, we could improve the
situation by providing a Lit configuration that allows compiling (but
not running) all the tests, building the tests on a supported macOS, and
then shipping those tests on an older backdeployment target in order to
run them against the system library. Since that requires significant
engineering, this isn't done at this time.
Without this patch `basic_string` cannot be properly resized to be
`max_size()` elements in size, even if an allocation is successful.
`__grow_by` allocates one less element than required, resulting in an
out-of-bounds access. At the same time, `max_size()` has an off-by-one
error, since there has to be space to store the null terminator, which
is currently ignored.
The current implementation of the `shrink_to_fit()` function of
`basic_string` swaps to the newly allocated buffer when the new buffer
has the same capacity as the existing one. While this is not incorrect,
it is truly unnecessary to swap to an equally-sized buffer. With equal
capacity, we should keep using the existing buffer and simply deallocate
the new one, avoiding the extra work of copying elements.
The desired behavior was documented in the following comment within the
function:
61ad08792a/libcxx/include/string (L3560-L3566)
However, the existing implementation did not exactly conform to this
guideline, which is a QoI matter.
This PR modifies the `shrink_to_fit()` function to ensure that the
buffer is only swapped when the new allocation is strictly smaller than
the existing one. When the capacities are equal, the new buffer will be
discarded without copying the elements. This is achieved by including
the `==` check in the above conditional logic.
The capacity is now passed correctly and a test for this path is added.
Since we changed the implementation of `reserve(size_type)` to only ever
extend,
it doesn't make a ton of sense anymore to have `__shrink_or_extend`,
since the code
paths of `reserve` and `shrink_to_fit` are now almost completely
separate.
This patch splits up `__shrink_or_extend` so that the individual parts
are in `reserve`
and `shrink_to_fit` depending on where they are needed.
This reverts commit 59f57be94f38758616b1339b293b43af845571af.
The increasing_allocator<T> class, originally introduced to test shrink_to_fit()
for std::vector, std::vector<bool>, and std::basic_string, has duplicated
definitions across several test files. Given the potential utility of this
class for capacity-related tests in various sequence containers, this patch
refactors the definition of increasing_allocator<T> into a single, reusable
location.
Making it meet the requirements for allocator since C++11. Fixes
#113609.
This PR doesn't make it meet the C++03 allocator requirements, because
that would make the type too verbose and libc++ has backported many
C++11 features to the C++03 mode.
Drive-by: Removes the `TEST_CONSTEXPR_CXX14` on `allocate`/`dealocate`
which is never in effect (and causes IFNDR-ness before C++23), since
these functions modify the namespace-scoped variable `allocated_`.
This is regression test for #90292.
Allocator used in test is very similar to test_allocator.
However, reproducer requires size_type of the string
to be 64bit, but test_allocator uses 32bit.
32bit size_type makes `sizeof(string::__long)` to be 16,
but the alignment issue fixed with #90292 is only triggered
with default `sizeof(string::__long)` which is 24.
Fixes#92128.
---------
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
~~NB: This PR depends on #78876. Ignore the first commit when reviewing,
and don't merge it until #78876 is resolved. When/if #78876 lands, I'll
clean this up.~~
This partially restores parity with the old, since removed debug build.
We now can re-enable a bunch of the disabled tests. Some things of note:
- `bounded_iter`'s converting constructor has never worked. It needs a
friend declaration to access the other `bound_iter` instantiation's
private fields.
- The old debug iterators also checked that callers did not try to
compare iterators from different objects. `bounded_iter` does not
currently do this, so I've left those disabled. However, I think we
probably should add those. See
https://github.com/llvm/llvm-project/issues/78771#issuecomment-1902999181
- The `std::vector` iterators are bounded up to capacity, not size. This
makes for a weaker safety check. This is because the STL promises not to
invalidate iterators when appending up to the capacity. Since we cannot
retroactively update all the iterators on `push_back()`, I've instead
sized it to the capacity. This is not as good, but at least will stop
the iterator from going off the end of the buffer.
There was also no test for this, so I've added one in the `std`
directory.
- `std::string` has two ambiguities to deal with. First, I opted not to
size it against the capacity. https://eel.is/c++draft/string.require#4
says iterators are invalidated on an non-const operation. Second,
whether the iterator can reach the NUL terminator. The previous debug
tests and the special-case in https://eel.is/c++draft/string.access#2
suggest no. If either of these causes widespread problems, I figure we
can revisit.
- `resize_and_overwrite.pass.cpp` assumed `std::string`'s iterator
supported `s.begin().base()`, but I see no promise of this in the
standard. GCC also doesn't support this. I fixed the test to use
`std::to_address`.
- `alignof.compile.pass.cpp`'s pointer isn't enough of a real pointer.
(It needs to satisfy `NullablePointer`, `LegacyRandomAccessIterator`,
and `LegacyContiguousIterator`.) `__bounded_iter` seems to instantiate
enough to notice. I've added a few more bits to satisfy it.
Fixes#78805
This patch removes many annotations that are not relevant anymore since
we don't support or test back-deploying to macOS < 10.13. It also cleans
up raw usage of target triples to identify versions of dylibs shipped on
prior versions of macOS, and uses the target-agnostic Lit features
instead. Finally, it reorders both the Lit backdeployment features and
the corresponding availability macros in the library in a way that makes
more sense, and reformulates the Lit backdeployment features in terms of
when a version of LLVM was introduced instead of encoding the system
versions on which it hasn't been introduced yet. Although one can be
derived from the other, encoding the negative form is extremely
error-prone.
Fixes#80901
We were not making any distinction between e.g. the "Apple-flavored"
libc++ built from trunk and the system-provided standard library on
Apple platforms. For example, any test that would be XFAILed on a
back-deployment target would unexpectedly pass when run on that
deployment target against the tip of trunk Apple-flavored libc++. In
reality, that test would be expected to pass because we're running
against the latest libc++, even if it is Apple-flavored.
To solve this issue, we introduce a new feature that describes whether
the Standard Library in use is the one provided by the system by
default, and that notion is different from the underlying standard
library flavor. We also refactor the existing Lit features to make a
distinction between availability markup and the library we're running
against at runtime, which otherwise limit the flexibility of what we can
express in the test suite. Finally, we refactor some of the
back-deployment versions that were incorrect (such as thinking that LLVM
10 was introduced in macOS 11, when in reality macOS 11 was synced with
LLVM 11).
Fixes#82107
Unconditionally change std::string's alignment to 8.
This change saves memory by providing the allocator more freedom to
allocate the most
efficient size class by dropping the alignment requirements for
std::string's
pointer from 16 to 8. This changes the output of std::string::max_size,
which makes it ABI breaking.
That said, the discussion concluded that we don't care about this ABI
break. and would like this change enabled universally.
The ABI break isn't one of layout or "class size", but rather the value
of "max_size()" changes, which in turn changes whether `std::bad_alloc`
or `std::length_error` is thrown for large allocations.
This change is the child of PR #68807, which enabled the change behind
an ABI flag.
This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.
**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.
To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.
Originally proposed here: https://reviews.llvm.org/D132769
Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214
This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.
The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.
This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.
Due to the functionality introduced in
[D132522](dd1b7b797a),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.
If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com
Extend `std::basic_string` tests to cover more buffer situations and
length in general, particularly non-SSO cases after SSO test cases
(changing buffers). This commit is a side effect of working on tests for
ASan annotations.
Related PR: https://github.com/llvm/llvm-project/pull/72677
While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked basic test
coverage like testing std::string itself (e.g. the test was only checking
std::basic_string with a custom allocator). In these cases, I did a few
conservative drive-by changes.
Differential Revision: https://reviews.llvm.org/D140550
Co-authored-by: Brendan Emery <brendan.emery@esrlabs.com>
The use_system_cxx_lib Lit feature was only used for back-deployment
testing. However, one immense hole in that setup was that we didn't
have a proper way to test Apple's own libc++ outside of back-deployment,
which was embodied by the fact that we needed to define _LIBCPP_DISABLE_AVAILABILITY
when testing (see change in libcxx/utils/libcxx/test/params.py).
This led to the apple-system testing configuration not checking for
availability markup, which is obviously quite bad since the library
we ship actually has availability markup.
Using stdlib=<VENDOR>-libc++ instead to encode back-deployment restrictions
on tests is simpler and it makes it possible to naturally support tests
such as availability markup checking even in the tip-of-trunk Apple-libc++
configuration.
Differential Revision: https://reviews.llvm.org/D146366
This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)size_t)|\1std::\2|' \{} \;
And manually removed some false positives in std/depr/depr.c.headers.
The `std` module doesn't export `::size_t`, this is a preparation for that module.
Reviewed By: ldionne, #libc, EricWF, philnik
Differential Revision: https://reviews.llvm.org/D146088
We pretty consistently don't define those cause they are not needed,
and it removes the potential pitfall to think that these tests are
being run. This doesn't touch .compile.fail.cpp tests since those
should be replaced by .verify.cpp tests anyway, and there would be
a lot to fix up.
As a fly-by, I also fixed a bit of formatting, removed a few unused
includes and made some very minor, clearly NFC refactorings such as
in allocator.traits/allocator.traits.members/allocate.verify.cpp where
the old test basically made no sense the way it was written.
Differential Revision: https://reviews.llvm.org/D146236
This patch switches the build compiler for AIX from ibm-clang to clang. ibm-clang++_r has `-pthread` by default, but clang for AIX doesn't, so `-pthread` had to be added to the test config. A bunch of tests now pass, so the `XFAIL` was removed. This patch also switch the build to use the visibility support available in clang-15 to control symbols exported by the shared library (AIX traditionally uses explicit export lists for this purpose).
Reviewed By: #libc, #libc_abi, daltenty, #libunwind, ldionne
Differential Revision: https://reviews.llvm.org/D127470
Since those features are general properties of the environment, it makes
sense to use them from libc++abi too, and so the name libcpp-has-no-xxx
doesn't make sense.
Differential Revision: https://reviews.llvm.org/D126482
This makes the code a bit simpler and (I think) removes the undefined behaviour from the normal string layout.
Reviewed By: ldionne, Mordante, #libc
Spies: labath, dblaikie, JDevlieghere, krytarowski, jgorbe, jingham, saugustine, arichardson, libcxx-commits
Differential Revision: https://reviews.llvm.org/D123580
These are the last™ changes to the tests for constexpr preparation.
Reviewed By: Quuxplusone, #libc, Mordante
Spies: Mordante, EricWF, libcxx-commits
Differential Revision: https://reviews.llvm.org/D120951
... from testing with MSVC's STL. Mostly truncation warnings and variables that are only used in `LIBCPP_ASSERT`.
Differential Revision: https://reviews.llvm.org/D116878
As explained in https://stackoverflow.com/a/70339311/627587, the fact
that shrink_to_fit wasn't defined as inline lead to issues when explicitly
instantiating basic_string. While explicit instantiations are always
somewhat brittle, this one was clearly a bug on our end.
Differential Revision: https://reviews.llvm.org/D115656
Make test_allocator etc. constexpr-friendly so they can be used to test constexpr string and possibly constexpr vector
Reviewed By: Quuxplusone, #libc, ldionne
Differential Revision: https://reviews.llvm.org/D110994
Even if these comments have a benefit in .h files (for editors that
care about language but can't be configured to treat .h as C++ code),
they certainly have no benefit for files with the .cpp extension.
Discussed in D110794.
Now that Lit supports regular expressions inside XFAIL & friends, it is
much easier to write Lit annotations based on the triple.
Differential Revision: https://reviews.llvm.org/D104747
This fixes a long standing issue where the triple is not always set
consistently in all configurations. This change also moves the
back-deployment Lit features to using the proper target triple
instead of using something ad-hoc.
This will be necessary for using from scratch Lit configuration files
in both normal testing and back-deployment testing.
Differential Revision: https://reviews.llvm.org/D102012
C++98 and C++03 are effectively aliases as far as Clang is concerned.
As such, allowing both std=c++98 and std=c++03 as Lit parameters is
just slightly confusing, but provides no value. It's similar to allowing
both std=c++17 and std=c++1z, which we don't do.
This was discovered because we had an internal bot that ran the test
suite under both c++98 AND c++03 -- one of which is redundant.
Differential Revision: https://reviews.llvm.org/D80926
Tests that require support for Clang-verify are already marked as such
explicitly by their extension, which is .verify.cpp. Requiring the use
of an explicit Lit feature is, after thought, not really helpful.
This is a change in design: we have been bitten in the past by tests not
being enabled when we thought they were. However, the issue was mostly
with file extensions being ignored. The fix for that is not to blindly
require explicit features all the time, but instead to report all files
that are in the suite but that don't match any known test format. This
can be implemented in a follow-up patch.
Instead of having different names for the same Lit feature accross code
bases, use the same name everywhere. This NFC commit is in preparation
for a refactor where all three projects will be using the same Lit
feature detection logic, and hence it won't be convenient to use
different names for the feature.
Differential Revision: https://reviews.llvm.org/D78370
The libc++ test suite has a lot of old Lit features used to XFAIL tests
and mark them as UNSUPPORTED. Many of them are to workaround problems on
old compilers or old platforms. As time goes by, it is good to go and
clean those up to simplify the configuration of the test suite, and also
to reflect the testing reality. It's not useful to have markup that gives
the impression that e.g. clang-3.3 is supported, when we don't really
test on it anymore (and hence several new tests probably don't have the
necessary markup on them).
Forcing -Werror and other warnings means that the test suite isn't
actually testing what most people are seeing in their code -- it seems
better and less arbitrary to compile these tests as close as possible
to the compiler default instead.
Removing -Werror also means that we get to differentiate between
diagnostics that are errors and those that are warnings, which makes
the test suite more precise.
Differential Revision: https://reviews.llvm.org/D76311
Some tests do not fail at all when -verify is not supported, unless some
arbitrary warning flag is added to make them fail. We currently used
-Werror=unused-result to make them fail, but doing so makes the test
suite a lot more inscrutable. It seems better to just disable those
tests when -verify is not supported.
Differential Revision: https://reviews.llvm.org/D76256
