This patch adds a large number of missing includes in the libc++ headers
and the test suite. Those were found as part of the effort to move
towards a mostly monolithic top-level std module.
See [LWG4061](https://cplusplus.github.io/LWG/issue4061) and
[P3341R0](https://wg21.link/p3341r0). Effectively reverts commit
36ce0c3b1e581ca310ae7d0cbc6af002cc5d0251.
`libcxx/test/std/utilities/format/format.functions/bug_81590.compile.pass.cpp`
has a `format` function that unexpectedly takes the
`basic_format_context` by value, which is made ill-formed by LWG4061.
This PR changes the function to take the context by reference.
The change increments the size of the lookup table considerably. The
table has an "upper boundary" check. The removal of the code units with
the property Grapheme_Extend=Yes removes the range E0100..E01EF. This
breaks the trailing large continuous section in two parts. This will be
improved in a followup patch.
Implements:
- P2713R1 Escaping improvements in std::format
- LWG3965 Incorrect example in [format.string.escaped] p3 for formatting
of combining characters
```
---------------------------------------------------------
Benchmark Before After
---------------------------------------------------------
BM_ascii_escaped<char> 95696 ns 110704 ns
BM_unicode_escaped<char> 89311 ns 101371 ns
BM_cyrillic_escaped<char> 58633 ns 63329 ns
BM_japanese_escaped<char> 44500 ns 41223 ns
BM_emoji_escaped<char> 99156 ns 111022 ns
BM_ascii_escaped<wchar_t> 92245 ns 112441 ns
BM_unicode_escaped<wchar_t> 80970 ns 102776 ns
BM_cyrillic_escaped<wchar_t> 51253 ns 58977 ns
BM_japanese_escaped<wchar_t> 37252 ns 36885 ns
BM_emoji_escaped<wchar_t> 96226 ns 115885 ns
```
As @cpplearner explained in microsoft/STL#4328:
> libc++'s "ascii" mode (controlled by the `_LIBCPP_HAS_NO_UNICODE`
> macro) means "every code unit outside ASCII is treated as a valid
> printable character". AFAIK we \[MSVC's STL\] don't support such a mode.
Because these files are testing a non-Standard mode, they should be
moved from `libcxx/test/std` to `libcxx/test/libcxx`.
Found while running libc++'s test suite with MSVC's STL.
* In `escaped_output.unicode.pass.cpp`, replace `_LIBCPP_SHORT_WCHAR`
with `TEST_SHORT_WCHAR`.
+ This was the only test that was directly using the `_LIBCPP` macro.
`libcxx/test/support/test_macros.h` performs this mapping:
c60ac50939/libcxx/test/support/test_macros.h (L442-L444)
* In `msvc_stdlib_force_include.h`, define `TEST_SHORT_WCHAR`.
I've structured this into a series of commits for even easier reviewing,
if that helps. I could easily split this up into separate PRs if
desired, but as this is low-risk with simple edits, I thought one PR
would be easiest.
* Drop unnecessary semicolons after function definitions.
* Cleanup comment typos.
* Cleanup `static_assert` typos.
* Cleanup test code typos.
+ There should be no functional changes, assuming I've changed all
occurrences.
* ~~Fix massive test code typos.~~
+ This was a real problem, but needed more surgery. I reverted those
changes here, and @philnik777 is fixing this properly with #73444.
* clang-formatting as requested by the CI.
Mark tests as necessary to accommodate Android L (5.0 / API 21) and up.
Add three Android lit features:
- android
- android-device-api=(21,22,23,...)
- LIBCXX-ANDROID-FIXME (for failures that need follow-up work)
Enable an AIX workaround in filesystem_test_helper.h for the broken
chmod on older Android devices.
Mark failing test with XFAIL or UNSUPPORTED:
- Mark modules tests as UNSUPPORTED, matching other configurations.
- Mark a gdb test as UNSUPPORTED.
- XFAIL tests for old devices that lack an API (fmemopen).
- XFAIL various FS tests (because SELinux blocks FIFO and hard linking,
because fchmodat is broken on old devices).
- XFAIL various locale tests (because Bionic has limited locale
support). (Also XFAIL an re.traits test.)
- XFAIL some print.fun tests because the error exception has no system
error string.
- Mark std::{cin,wcin} tests UNSUPPORTED because they hang with
adb_run.py on old devices.
- Mark a few tests UNSUPPORTED because they allocate too much memory.
- notify_one.pass.cpp is flaky on Android.
- XFAIL libc++abi demangler test because of Android's special long
double on x86[-64].
N.B. The `__ANDROID_API__` macro specifies a minimum required API level
at build-time, whereas the android-device-api lit feature is the
detected API level of the device at run-time. The android-device-api
value will be >= `__ANDROID_API__`.
This commit was split out from https://reviews.llvm.org/D139147.
Fixes: https://github.com/llvm/llvm-project/issues/69270
The test is hardcoded to fail after passing `test_ill_formed_utf16()`. It passes on 32-bit AIX if we remove this.
Reviewed By: Mordante, #libc, ldionne
Differential Revision: https://reviews.llvm.org/D150273
Improves both the compile-time and run-time errors.
At compile-time it does a bit more work to get more specific errors.
This could be done at run-time too, but that has a performance penalty.
Since it's expected most use-cases use format* instead of vformat* the
compile-time errors are more common.
For example when using
std::format_to("{:-c}", 42);
Before compile output would contain
std::__throw_format_error("The format-spec should consume the input or end with a '}'");
Now it contains
std::__throw_format_error("The format specifier does not allow the sign option");
Given a better indication the sign option is not allowed. Note the
output is still not user-friendly; C++ doesn't have good facilities to
generate nice messages from the library.
In general all messages have been reviewed and improved, using a more
consistent style and using less terms used in the standard. For example
format-spec -> format specifier
arg-id -> argument index
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D152624
After parsing a std-format-spec it's validated, depending on the type used some
format options are not allowed. This improves the error messages in the
exceptions thrown upon failure.
Depends on D155364
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D155366
Drive-by fix to make sure the __retarget_buffer works correctly whan
using a hint of 1. This was discovered in one of the new tests.
Drive-by fixes __retarget_buffer when initialized with size 1.
Implements parts of
- P2093R14 Formatted output
- P2539R4 Should the output of std::print to a terminal be
synchronized with the underlying stream?
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D150044
The post-condition on the functions is that the buffer is not full.
This post-conditon is used as pre-condition of the push_back function.
When a copy, fill, of transform function exactly fit in the buffer this
post-condition was validated.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D155397
The feature is applied as DR instead of a normal paper. MSVC STL and
libstdc++ will do the same.
Implements
- P2510R3 Formatting pointers
Depends on D153192
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D153195
The mask used to check whether a code unit is a valid continuation was
incorrect and accepts non-continuation code points. This fixes the
issue.
Reviewed By: ldionne, tahonermann, #libc
Differential Revision: https://reviews.llvm.org/D149672
Some tests in our test suite are unbelievably slow on GCC due to the
use of the always_inline attribute. See [1] for more details.
This patch introduces the GCC-ALWAYS_INLINE-FIXME lit feature to
disable tests that are plagued by that issue. At the same time, it
moves several existing tests from ad-hoc `UNSUPPORTED: gcc-12` markup
to the new GCC-ALWAYS_INLINE-FIXME feature, and marks the slowest tests
reported by the CI as `UNSUPPORTED: GCC-ALWAYS_INLINE-FIXME`.
[1]: https://discourse.llvm.org/t/rfc-stop-supporting-extern-instantiations-with-gcc/71277/1
Differential Revision: https://reviews.llvm.org/D152736
This is an extension and only adds the functions that are a considered a
but when called and ignoring the result.
Drive-by sort all nodiscard extensions in the documentation.
Reviewed By: #libc, philnik
Differential Revision: https://reviews.llvm.org/D152097
The code has been quite ready for a while now and there are no more ABI
breaking papers. So this is a good time to mark the feature as stable.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D150802
The main change is to allow a UCS scalar value as fill character.
Especially for char based formatting this increase the number of valid
characters. Originally this was to be expected ABI breaking, however the
current change does not seem to break the ABI.
Implements
- P2572 std::format() fill character allowances
Depends on D144499
Reviewed By: ldionne, tahonermann, #libc
Differential Revision: https://reviews.llvm.org/D144742
escaped_output.unicode.pass.cpp is failing only on 32-bit AIX. The rest are passing.
Reviewed by: #libc, Mordante
Differential Revision: https://reviews.llvm.org/D149078
As obvious from the paper's title this is an LWG issue and thus retroactively
applied to C++20. This change may the output for certain code points:
1 Considers 8477 extra codepoints as having a width 2 (as of Unicode 15)
(mostly Tangut Ideographs)
2 Change the width of 85 unassigned code points from 2 to 1
3 Change the width of 8 codepoints (in the range U+3248 CIRCLED NUMBER
TEN ON BLACK SQUARE ... U+324F CIRCLED NUMBER EIGHTY ON BLACK
SQUARE) from 2 to 1, because it seems questionable to make an exception
for those without input from Unicode
Note that libc++ already uses Unicode 15, while the Standard requires Unicode 12.
(The last time I checked MSVC STL used Unicode 14.)
So in practice the only notable change is item 3.
Implements
P2675 LWG3780: The Paper
format's width estimation is too approximate and not forward compatible
Benchmark before these changes
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_ascii_text<char> 3928 ns 3928 ns 178131
BM_unicode_text<char> 75231 ns 75230 ns 9158
BM_cyrillic_text<char> 59837 ns 59834 ns 11529
BM_japanese_text<char> 39842 ns 39832 ns 17501
BM_emoji_text<char> 3931 ns 3930 ns 177750
BM_ascii_text<wchar_t> 4024 ns 4024 ns 174190
BM_unicode_text<wchar_t> 63756 ns 63751 ns 11136
BM_cyrillic_text<wchar_t> 44639 ns 44638 ns 15597
BM_japanese_text<wchar_t> 34425 ns 34424 ns 20283
BM_emoji_text<wchar_t> 3937 ns 3937 ns 177684
Benchmark after these changes
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_ascii_text<char> 3914 ns 3913 ns 178814
BM_unicode_text<char> 70380 ns 70378 ns 9694
BM_cyrillic_text<char> 51889 ns 51877 ns 13488
BM_japanese_text<char> 41707 ns 41705 ns 16723
BM_emoji_text<char> 3908 ns 3907 ns 177912
BM_ascii_text<wchar_t> 3949 ns 3948 ns 177525
BM_unicode_text<wchar_t> 64591 ns 64587 ns 10649
BM_cyrillic_text<wchar_t> 44089 ns 44078 ns 15721
BM_japanese_text<wchar_t> 39369 ns 39367 ns 17779
BM_emoji_text<wchar_t> 3936 ns 3934 ns 177821
Benchmarks without "if(__code_point < (__entries[0] >> 14))"
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_ascii_text<char> 3922 ns 3922 ns 178587
BM_unicode_text<char> 94474 ns 94474 ns 7351
BM_cyrillic_text<char> 69202 ns 69200 ns 10157
BM_japanese_text<char> 42735 ns 42692 ns 16382
BM_emoji_text<char> 3920 ns 3919 ns 178704
BM_ascii_text<wchar_t> 3951 ns 3950 ns 177224
BM_unicode_text<wchar_t> 81003 ns 80988 ns 8668
BM_cyrillic_text<wchar_t> 57020 ns 57018 ns 12048
BM_japanese_text<wchar_t> 39695 ns 39687 ns 17582
BM_emoji_text<wchar_t> 3977 ns 3976 ns 176479
This optimization does carry its weight for the Unicode and Cyrillic
test. For the Japanese tests the gains are minor and for emoji it seems
to have no effect.
Reviewed By: ldionne, tahonermann, #libc
Differential Revision: https://reviews.llvm.org/D144499
LWG3720 Restrict the valid types of arg-id for width and precision in
std-format-spec
Depends on D144325
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D144326
Instead of writing something like `XFAIL: use_system_cxx_lib && target=...`
to XFAIL back-deployment tests, introduce named Lit features like
`availability-shared_mutex-missing` to represent those. This makes the
XFAIL annotations leaner, and solves the problem of XFAIL comments
potentially getting out of sync. This would also make it easier for
another vendor to add their own annotations to the test suite by simply
changing how the feature is defined for their OS releases, instead
of having to modify hundreds of tests to add repetitive annotations.
This doesn't touch *all* annotations -- only annotations that were widely
duplicated are given named features (e.g. when filesystem or shared_mutex
were introduced). I still think it probably doesn't make sense to have a
named feature for every single fix we make to the dylib.
This is in essence a revert of 2659663, but since then the test suite
has changed significantly. Back when I did 2659663, the configuration
files we have for the test suite right now were being bootstrapped and
it wasn't clear how to provide these features for back-deployment in
that context. Since then, we have a streamlined way of defining these
features in `features.py` and that doesn't impact the ability for a
configuration file to stay minimal.
The original motivation for this change was that I am about to propose
a change that would touch essentially all XFAIL annotations for back-deployment
in the test suite, and this greatly reduces the number of lines changed
by that upcoming change, in addition to making the test suite generally
better.
Differential Revision: https://reviews.llvm.org/D146359
Those seem to have been failing for a while but we might not have noticed
because of the recent CI instability issues. I'm marking them as unsupported
to try to get the CI functional again, especially since the majority of
<format> tests are already not working on GCC 12.
std::format is currently experimental, so there is technically no
deployment target requirement for it (since the only symbols required
for it are in `libc++experimental.a`).
However, some parts of std::format depend indirectly on the floating
point std::to_chars implementation, which does have deployment target
requirements.
This patch removes all the availability format for std::format and
updates the XFAILs in the tests to properly explain why they fail
on old deployment targets, when they do. It also changes a couple
of tests to avoid depending on floating-point std::to_chars when
it isn't fundamental to the test.
Finally, some tests are marked as XFAIL but I added a comment saying
TODO FMT This test should not require std::to_chars(floating-point)
These tests do not fundamentally depend on floating-point std::to_chars,
however they end up failing because calling std::format even without a
floating-point argument to format will end up requiring floating-point
std::to_chars. I believe this is an implementation artifact that could
be avoided in all cases where we know the format string at compile-time.
In the tests, I added the TODO comment only to the places where we could
do better and actually avoid relying on floating-point std::to_chars
because we know the format string at compile-time.
Differential Revision: https://reviews.llvm.org/D134598
This has been done using the following commands
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)ptrdiff_t)|\1std::\2|' \{} \;
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)max_align_t)|\1std::\2|' \{} \;
The std module doesn't export declarations in the global namespaace.,
This is a preparation for that module.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D146550
This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)size_t)|\1std::\2|' \{} \;
And manually removed some false positives in std/depr/depr.c.headers.
The `std` module doesn't export `::size_t`, this is a preparation for that module.
Reviewed By: ldionne, #libc, EricWF, philnik
Differential Revision: https://reviews.llvm.org/D146088
This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)(?<!::u)u?int(_[a-z]+)?[0-9]{1,2}_t)|\1std::\2|' \{} \;
And manually removed some false positives in std/depr/depr.c.headers.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D145880
During the implementation of P2286 a second Unicode decoder was added.
The original decoder was only used for the width estimation. Changing
an ill-formed Unicode sequence to the replacement character, works
properly for this use case. For P2286 an ill-formed Unicode sequence
needs to be formatted as a sequence of code units. The exact wording in
the Standard as a bit unclear and there was odd example in the WP. This
made it hard to use the same decoder. SG16 determined the odd example in
the WP was a bug and this has been fixed in the WP.
This made it possible to combine the two decoders. The P2286 decoder
kept track of the size of the ill-formed sequence. However this was not
needed since the output algorithm needs to keep track of size of a
well-formed and an ill-formed sequence. So this feature has been
removed.
The error status remains since it's needed for P2286, the grapheme
clustering can ignore this unneeded value. (In general, grapheme
clustering is only has specified behaviour for Unicode. When the string
is in a non-Unicode encoding there are no requirements. Ill-formed
Unicode is a non-Unicode encoding. Still libc++ does a best effort
estimation.)
There UTF-8 decoder accepted several ill-formed sequences:
- Values in the surrogate range U+D800..U+DFFF.
- Values encoded in more code units than required, for example 0+0020
in theory can be encoded using 1, 2, 3, or 4 were accepted. This is
not allowed by the Unicode Standard.
- Values larger than U+10FFFF were not always rejected.
Reviewed By: #libc, ldionne, tahonermann, Mordante
Differential Revision: https://reviews.llvm.org/D144346
The module std does not provide c-types in the global namespace. This
means all these types need to be fully qualified. This is a first step
to convert them by using sed.
Since this is an automated conversion other types like uint64_t are kept
as is.
Note that tests in the directory libcxx/test/std/depr/depr.c.headers
should not be converted automatically. This requires manual attention,
there some test require testing uint32_t in the global namespace. These
test should fail when using the std module, and pass when using the
std.compat module.
A similar issue occurs with atomic, atomic_uint32_t is specified as
using atomic_uint32_t = atomic<uint32_t>; // freestanding
So here too we need to keep the name in the global namespace in the
tests.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D145520
Fixes llvm.org/PR58714 reported by @jwakely and a similar issue
reported privately by @vitaut.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D145306
These macros are intended to replace the macros in rapid-cxx-test.h.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D142808
Using some builds the modular build fails due to missing exports
and includes. This fixes the build.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D143203
The constexpr validation parsed parts of the format string that didn't
belong to the specific replacement field.
Fixes https://llvm.org/PR60536
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D143402
These macros make it easier to log additional information. This is
useful for formatting tests. It also properly disables additional
information when locales are disabled in libc++.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D140651
This reverts commit a6e1080b87db8fbe0e1afadd96af5a3c0bd5e279.
Fix the conditions when the `memmove` optimization can be applied and refactor them out into a reusable type trait, fix and significantly expand the tests.
Differential Revision: https://reviews.llvm.org/D139235
The format function test serve two purposes:
- Test whether all format functions work in general.
- Test whether all formatting rules are implemented correctly.
At the moment the *pass.cpp tests do both. These tests are quite slow,
while testing all rules for all functions doesn't add much coverage.
There are two execution modi of the format functions:
- run-time validation in the vformat functions.
- compile-time validation in the other function.
So instead of running all tests for all functions, they are only used for
format.pass.cpp and vformat.pass.cpp still do all tests.
The other tests do a smaller set of test, just to make sure they work in the
basics.
Running the format tests using one thread:
- before 00:04:16
- after 00:02:14
The slow tests were also reported in
https::llvm.org/PR58141
Also split a generic part of the test to a generic support header. This
allows these parts to be reused in the range-based formatter tests.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D140115
Per our policy, the latest released AppleClang has been 14 for a while,
so libc++ is removing support for AppleClang 13. Our CI bots have been
moved to AppleClang 14 a few weeks ago.
Differential Revision: https://reviews.llvm.org/D138685
LWG-3539 was already implemented but not marked as done.
LWG-3567 is implemented in this commit.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D112368
This is mainly to improve the readability of the tests. As a side
effects the tests run faster too,
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D135288
