The `grouping` string for locale `en_US.UTF-8` and `fr_FR.UTF-8` on AIX
is `3`. This is different from Linux's `3;3` but is the same as Windows.
This patch removes `XFAIL: LIBCXX-AIX-FIXME` and changes to use the
`WIN32` code path.
The C locale on AIX uses `ISO-8859-1`, where `0xFB` is a valid
character. Widening char(-5) succeeds and produces L'\u00fb' the same as
on macOS, FreeBSD, and Windows. This patch removes `XFAIL:
LIBCXX-AIX-FIXME` and uses the macOS, FreeBSD, and WIN32 code path for
AIX.
`__apply_cv_t` and `__copy_cvref_t` are very closely related. They are
in fact identical except that `__copy_cvref_t` handles rvalue references
properly. Some uses don't actually require handling of references, so
they are replaced with `__copy_cv_t`.
The patch does several things:
- fixes module exports
- disables clang-tidy with Clang-17 due to known issues
- disabled clang-tidy on older libstdc++ versions since it lacks C++20
features used
- fixes the CMake dependency
The issue why clang-tidy was not used in the CI was the last issue; the
plugin was not a
dependency of the tests. Without a plugin the tests disable clang-tidy.
This was noticed while investigating
https://github.com/llvm/llvm-project/issues/89898
This pull request is the third iteration aiming to integrate short
string annotations. This commit includes:
- Enabling basic_string annotations for short strings.
- Setting a value of `__trivially_relocatable` in `std::basic_string` to
`false_type` when compiling with ASan (nothing changes when compiling
without ASan). Short string annotations make `std::basic_string` to not
be trivially relocatable, because memory has to be unpoisoned.
- Adding a `_LIBCPP_STRING_INTERNAL_MEMORY_ACCESS` modifier to two
functions.
- Creating a macro `_LIBCPP_ASAN_VOLATILE_WRAPPER` to prevent
problematic stack optimizations (the macro modifies code behavior only
when compiling with ASan).
Previously we had issues with compiler optimization, which we understand
thanks to @vitalybuka. This commit also addresses smaller changes in
short string, since previous upstream attempts.
Problematic optimization was loading two values in code similar to:
```
__is_long() ? __get_long_size() : __get_short_size();
```
We aim to resolve it with the volatile wrapper.
This commit is built on top of two previous attempts which descriptions
are below.
Additionally, in the meantime, annotations were updated (but it
shouldn't have any impact on anything):
- https://github.com/llvm/llvm-project/pull/79292
---
Previous PR: https://github.com/llvm/llvm-project/pull/79049
Reverted:
a16f81f5e3
Previous description:
Originally merged here: https://github.com/llvm/llvm-project/pull/75882
Reverted here: https://github.com/llvm/llvm-project/pull/78627
Reverted due to failing buildbots. The problem was not caused by the
annotations code, but by code in the `UniqueFunctionBase` class and in
the `JSON.h` file. That code caused the program to write to memory that
was already being used by string objects, which resulted in an ASan
error.
Fixes are implemented in:
- https://github.com/llvm/llvm-project/pull/79065
- https://github.com/llvm/llvm-project/pull/79066
Problematic code from `UniqueFunctionBase` for example:
```cpp
// In debug builds, we also scribble across the rest of the storage.
memset(RHS.getInlineStorage(), 0xAD, InlineStorageSize);
```
---
Original description:
This commit turns on ASan annotations in `std::basic_string` for short
stings (SSO case).
Originally suggested here: https://reviews.llvm.org/D147680
String annotations added here:
https://github.com/llvm/llvm-project/pull/72677
Requires to pass CI without fails:
- https://github.com/llvm/llvm-project/pull/75845
- https://github.com/llvm/llvm-project/pull/75858
Annotating `std::basic_string` with default allocator is implemented in
https://github.com/llvm/llvm-project/pull/72677 but annotations for
short strings (SSO - Short String Optimization) are turned off there.
This commit turns them on. This also removes
`_LIBCPP_SHORT_STRING_ANNOTATIONS_ALLOWED`, because we do not plan to
support turning on and off short string annotations.
Support in ASan API exists since
dd1b7b797a.
You can turn off annotations for a specific allocator based on changes
from
2fa1bec7a2.
This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.
The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.
If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com
Lit already has support for stopping LIT from parsing further test
directives. It is
// END.
After that directive, LIT will stop parsing.
This change removes the BLOCKLIT hack and replaces it with END.
Adjust some of the [rand.dist] critical values that are too strict
- Most critical values are determined empirically by running each test
51
times with a different PRNG seed and finding the smallest symmetric
interval
around the median that contains 90% of the sample means, variances, etc.
- For the Kolmogorov-Smirnov tests, the alpha=0.1 critical value for
large N
is 1.224/sqrt(N).
- For normally distributed variates, the sample kurtosis is distributed
as
Normal(0, 24/N). For N=1e5, this gives a 90% confidence interval of
0+/-0.0255. For Binomial(40, 0.25), which is approximately normal, the
kurtosis is -0.0167, so the relative 90% CI is large, on the order of
0.0255/0.0167 = 153%. In most cases the distribution of the sample
kurtosis
isn't known analytically, but similarly large relative tolerances can be
expected if the kurtosis is near zero.
This attribute is used in the headers. Not using this in the modules has
led to several issues. Add them to the modules to avoid these errors in
other placed.
This is detected by asan after #83774
Allocation size will be divided by `__endian_factor` before storing. If
it's not aligned,
we will not be able to recover allocation size to pass into
`__alloc_traits::deallocate`.
we have code like this
```
auto __allocation = std::__allocate_at_least(__alloc(), __recommend(__sz) + 1);
__p = __allocation.ptr;
__set_long_cap(__allocation.count);
void __set_long_cap(size_type __s) _NOEXCEPT {
__r_.first().__l.__cap_ = __s / __endian_factor;
__r_.first().__l.__is_long_ = true;
}
size_type __get_long_cap() const _NOEXCEPT {
return __r_.first().__l.__cap_ * __endian_factor;
}
inline ~basic_string() {
__annotate_delete();
if (__is_long())
__alloc_traits::deallocate(__alloc(), __get_long_pointer(), __get_long_cap());
}
```
1. __recommend() -> even size
2. `std::__allocate_at_least(__alloc(), __recommend(__sz) + 1)` - > not
even size
3. ` __set_long_cap() `- > lose one bit of size for __endian_factor == 2
(see `/ __endian_factor`)
4. `__alloc_traits::deallocate(__alloc(), __get_long_pointer(),
__get_long_cap())` -> uses even size (see `__get_long_cap`)
I strongly suspect nobody ever used that macro since it wasn't very well
known. Furthermore, it only affects a handful of diagnostics and I think
it makes sense to either provide them unconditionally, or to not
provided them at all.
This is needed for a workaround to make sure the link later succeeds. I
don't know the reason for that but it is definitely needed.
https://github.com/llvm/llvm-project/pull/89234 will/wants to correct
the triple normalisation for -none- and this means that clang prior to
19, and clang 19 and above will have different answers and therefore
different library paths.
I don't want to bootstrap a clang just for libcxx CI, or require that
anyone building for Arm do the same, so ask the compiler what the triple
should be.
This will be compatible with 17 and 19 when we do update to that
version.
I'm assuming $CC is what anyone locally would set to override the
compiler, and `cc` is the binary name in our CI containers. It's not
perfect but it should cover most use cases.
Implement
- LWG4053 Unary call to `std::views::repeat` does not decay the argument
- LWG4054 Repeating a `repeat_view` should repeat the view
Signed-off-by: yronglin <yronglin777@gmail.com>
The previous patch implemented
- P2713R1 Escaping improvements in std::format
- LWG3965 Incorrect example in [format.string.escaped] p3 for formatting
of combining characters
These changes were correct, but had a size and performance penalty. This
patch improves the size and performance of the previous patch. The
performance is still worse than before since the lookups may require two
property lookups instead of one before implementing the paper. The
changes give a tighter coupling between the Unicode data and the
algorithm. Additional tests are added to notify about changes in future
Unicode updates.
Before
```
-----------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------
BM_ascii_escaped<char> 110704 ns 110696 ns 6206
BM_unicode_escaped<char> 101371 ns 101374 ns 6862
BM_cyrillic_escaped<char> 63329 ns 63327 ns 11013
BM_japanese_escaped<char> 41223 ns 41225 ns 16938
BM_emoji_escaped<char> 111022 ns 111021 ns 6304
BM_ascii_escaped<wchar_t> 112441 ns 112443 ns 6231
BM_unicode_escaped<wchar_t> 102776 ns 102779 ns 6813
BM_cyrillic_escaped<wchar_t> 58977 ns 58975 ns 11868
BM_japanese_escaped<wchar_t> 36885 ns 36886 ns 18975
BM_emoji_escaped<wchar_t> 115885 ns 115881 ns 6051
```
The first change is to manually encode the entire last area and make a
manual exception for the 240 excluded entries. This reduced the table
from 1077 to 729 entries and gave the following benchmark results.
```
-----------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------
BM_ascii_escaped<char> 104777 ns 104776 ns 6550
BM_unicode_escaped<char> 96980 ns 96982 ns 7238
BM_cyrillic_escaped<char> 60254 ns 60251 ns 11670
BM_japanese_escaped<char> 44452 ns 44452 ns 15734
BM_emoji_escaped<char> 104557 ns 104551 ns 6685
BM_ascii_escaped<wchar_t> 107456 ns 107454 ns 6505
BM_unicode_escaped<wchar_t> 96219 ns 96216 ns 7301
BM_cyrillic_escaped<wchar_t> 56921 ns 56904 ns 12288
BM_japanese_escaped<wchar_t> 39530 ns 39529 ns 17492
BM_emoji_escaped<wchar_t> 108494 ns 108496 ns 6408
```
An entry in the table can only contain 2048 code points. For larger
ranges there are multiple entries split in chunks with a maximum size of
2048 entries. To encode the entire Unicode code point range 21 bits are
required. The manual part starts at 0x323B0 this means all entries in
the table fit in 18 bits. This allows to allocate 3 additional bits for
the range. This allows entries to have 16384 elements. This range always
avoids splitting the range in multiple chunks.
This reduces the number of table elements from 729 to 711 and gives the
following benchmark results.
```
-----------------------------------------------------------------------
Benchmark Time CPU Iterations
-----------------------------------------------------------------------
BM_ascii_escaped<char> 104289 ns 104289 ns 6619
BM_unicode_escaped<char> 96682 ns 96681 ns 7215
BM_cyrillic_escaped<char> 59673 ns 59673 ns 11732
BM_japanese_escaped<char> 41983 ns 41982 ns 16646
BM_emoji_escaped<char> 104119 ns 104120 ns 6683
BM_ascii_escaped<wchar_t> 104503 ns 104505 ns 6693
BM_unicode_escaped<wchar_t> 93426 ns 93423 ns 7489
BM_cyrillic_escaped<wchar_t> 54858 ns 54859 ns 12742
BM_japanese_escaped<wchar_t> 36385 ns 36384 ns 19259
BM_emoji_escaped<wchar_t> 105608 ns 105610 ns 6592
```
Primary motivation: is that after #84651 msan will
complain if fields accessed after ~__no_destroy.
My understanding of the https://eel.is/c++draft/basic.life#10
Static object with trivial destruction has program lifetime.
Static object with empty destuctor has implicit lifetime, and
accessing the object after lifetime is UB.
It was UB before #84651, it's just msan ignored union members.
Existing code with unions uses empty destructor, so accessing after
the main() can cause UB.
"placement new" version can have trivial destructor, so there is no end
of lifetime.
Secondary motivation: empty destructor will register __cxa_atexit with
-O0.
https://gcc.godbolt.org/z/hce587b65
We can not remove the destructor with union where
_Tp can have non-trivial destructor.
But we can remove destructor if we use in-place
new instead of union.
https://gcc.godbolt.org/z/Yqxx57eEd - empty even with -O0.
New test fails without the patch on
https://lab.llvm.org/buildbot/#/builders/sanitizer-x86_64-linux-bootstrap-msan
These macros are used by STL implementations to support implementation
of std::hardware_destructive_interference_size and
std::hardware_constructive_interference_size
Fixes#60174
---------
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
This pull request implements LWG3984: ranges::to's recursion branch
may be ill-formed.
In the current implementation, ranges::to's recursion branch pipes the
range into a `views::transform(/* lambda */)`, which is a __range_adaptor_closure
object. In libc++, the pipe operator of __range_adaptor_closure requires a
viewable_range, so the following code won't compile, as the type of lvalue
`r` doesn't model viewable_range:
#include <ranges>
#include <vector>
#include <list>
int main() {
std::vector<std::vector<int>> v;
auto r = std::views::all(std::move(v));
auto l = std::ranges::to<std::list<std::list<int>>>(r);
}
Co-authored-by: A. Jiang <de34@live.cn>
Since C++14 has been released for about nine years and most standard
libraries have implemented sized deallocation functions, it's time to
make this feature default again.
This is another try of https://reviews.llvm.org/D112921.
Fixes#60061
The change increments the size of the lookup table considerably. The
table has an "upper boundary" check. The removal of the code units with
the property Grapheme_Extend=Yes removes the range E0100..E01EF. This
breaks the trailing large continuous section in two parts. This will be
improved in a followup patch.
Implements:
- P2713R1 Escaping improvements in std::format
- LWG3965 Incorrect example in [format.string.escaped] p3 for formatting
of combining characters
```
---------------------------------------------------------
Benchmark Before After
---------------------------------------------------------
BM_ascii_escaped<char> 95696 ns 110704 ns
BM_unicode_escaped<char> 89311 ns 101371 ns
BM_cyrillic_escaped<char> 58633 ns 63329 ns
BM_japanese_escaped<char> 44500 ns 41223 ns
BM_emoji_escaped<char> 99156 ns 111022 ns
BM_ascii_escaped<wchar_t> 92245 ns 112441 ns
BM_unicode_escaped<wchar_t> 80970 ns 102776 ns
BM_cyrillic_escaped<wchar_t> 51253 ns 58977 ns
BM_japanese_escaped<wchar_t> 37252 ns 36885 ns
BM_emoji_escaped<wchar_t> 96226 ns 115885 ns
```
This patch implements LWG4023 by adding explicit assertions for the
added preconditions and also fixes a few tests that were violating these
preconditions.
To make it easier to maintain the availability macros for both upstream
developers and vendors, this patch reformulates availability macros as a
function of the upstream LLVM release that a feature was introduced in.
This way, upstream developers can easily use the appropriate LLVM
version, and vendors can simply fill in the platform version(s) in which
a LLVM version landed.
