422 Commits

Author	SHA1	Message	Date
Nikolas Klauser	fc5e0c071b	[libc++] Simplify most of the segmented iterator optimizations (#164797) ... This patch does two things. (1) It replaces SFINAE with `if constexpr`, avoiding some overload resolution and unnecessary boilerplate. (2) It removes an overload from `__for_each_n` to forward to `__for_each`, since `__for_each` doesn't provide any further optimizations.	2025-11-07 08:27:55 +01:00
Connector Switch	0621fd0b88	[libcxx] Optimize rng::generate_n for segmented iterators (#165280) ... Part of #102817. This patch optimizes `rng::generate_n` for segmented iterators by forwarding the implementation directly to `std::generate_n`. - before ``` rng::generate_n(deque<int>)/32 21.7 ns 22.0 ns 32000000 rng::generate_n(deque<int>)/50 30.8 ns 30.7 ns 22400000 rng::generate_n(deque<int>)/1024 492 ns 488 ns 1120000 rng::generate_n(deque<int>)/8192 3938 ns 3924 ns 179200 ``` - after ``` rng::generate_n(deque<int>)/32 11.0 ns 11.0 ns 64000000 rng::generate_n(deque<int>)/50 16.2 ns 16.1 ns 40727273 rng::generate_n(deque<int>)/1024 292 ns 286 ns 2240000 rng::generate_n(deque<int>)/8192 2291 ns 2302 ns 298667 ```	2025-10-28 22:22:26 +08:00
Connector Switch	abf80b54bc	[libc++][IWYU] Remove std::move header in std::for_each (#164272) ... It seems this was accidentally included; there's no use of std::move in this header.	2025-10-21 15:03:09 +00:00
Connector Switch	c06ae43e26	[libcxx] Optimize std::generate_n for segmented iterators (#164266) ... Part of #102817. This is a natural follow-up to #163006. We are forwarding `std::generate_n` to `std::__for_each_n` (`std::for_each_n` needs c++17), resulting in improved performance for segmented iterators. before: ``` std::generate_n(deque<int>)/32 17.5 ns 17.3 ns 40727273 std::generate_n(deque<int>)/50 25.7 ns 25.5 ns 26352941 std::generate_n(deque<int>)/1024 490 ns 487 ns 1445161 std::generate_n(deque<int>)/8192 3908 ns 3924 ns 179200 ``` after: ``` std::generate_n(deque<int>)/32 11.1 ns 11.0 ns 64000000 std::generate_n(deque<int>)/50 16.1 ns 16.0 ns 44800000 std::generate_n(deque<int>)/1024 291 ns 292 ns 2357895 std::generate_n(deque<int>)/8192 2269 ns 2250 ns 298667 ```	2025-10-21 12:01:36 +02:00
Connector Switch	46e8816928	[libcxx] Optimize std::generate for segmented iterators (#163006) ... Part of #102817. This patch attempts to optimize the performance of `std::generate` for segmented iterators. Below are the benchmark numbers from `libcxx\test\benchmarks\algorithms\modifying\generate.bench.cpp`. Test cases that use segmented iterators have also been added. - before ``` std::generate(deque<int>)/32 194 ns 193 ns 3733333 std::generate(deque<int>)/50 276 ns 276 ns 2488889 std::generate(deque<int>)/1024 5096 ns 5022 ns 112000 std::generate(deque<int>)/8192 40806 ns 40806 ns 17231 ``` - after ``` std::generate(deque<int>)/32 106 ns 105 ns 6400000 std::generate(deque<int>)/50 139 ns 138 ns 4977778 std::generate(deque<int>)/1024 2713 ns 2699 ns 248889 std::generate(deque<int>)/8192 18983 ns 19252 ns 37333 ``` --------- Co-authored-by: A. Jiang <de34@live.cn>	2025-10-20 19:37:33 +08:00
Peng Liu	d0cee6939a	[libc++] Optimize std::{,ranges}::{fill,fill_n} for segmented iterators (#132665) ... This patch optimizes `std::fill`, `std::fill_n`, `std::ranges::fill`, and `std::ranges::fill_n` for segmented iterators, achieving substantial performance improvements. Specifically, for `deque<int>` iterators, the performance improvements are above 10x for all these algorithms. The optimization also enables filling segmented memory of `deque<int>` to approach the performance of filling contiguous memory of `vector<int>`. Benchmark results comparing the before and after implementations are provided below. For additional context, we’ve included `vector<int>` results, which remain unchanged, as this patch specifically targets segmented iterators and leaves non-segmented iterator behavior untouched. Fixes two subtasks outlined in #102817. #### `fill_n` ``` ----------------------------------------------------------------------------- Benchmark Before After Speedup ----------------------------------------------------------------------------- std::fill_n(deque<int>)/32 11.4 ns 2.28 ns 5.0x std::fill_n(deque<int>)/50 19.7 ns 3.40 ns 5.8x std::fill_n(deque<int>)/1024 391 ns 37.3 ns 10.5x std::fill_n(deque<int>)/8192 3174 ns 301 ns 10.5x std::fill_n(deque<int>)/65536 26504 ns 2951 ns 9.0x std::fill_n(deque<int>)/1048576 407960 ns 80658 ns 5.1x rng::fill_n(deque<int>)/32 14.3 ns 2.15 ns 6.6x rng::fill_n(deque<int>)/50 20.2 ns 3.22 ns 6.3x rng::fill_n(deque<int>)/1024 381 ns 37.8 ns 10.1x rng::fill_n(deque<int>)/8192 3101 ns 294 ns 10.5x rng::fill_n(deque<int>)/65536 25098 ns 2926 ns 8.6x rng::fill_n(deque<int>)/1048576 394342 ns 78874 ns 5.0x std::fill_n(vector<int>)/32 1.76 ns 1.72 ns 1.0x std::fill_n(vector<int>)/50 3.00 ns 2.73 ns 1.1x std::fill_n(vector<int>)/1024 38.4 ns 37.9 ns 1.0x std::fill_n(vector<int>)/8192 258 ns 252 ns 1.0x std::fill_n(vector<int>)/65536 2993 ns 2889 ns 1.0x std::fill_n(vector<int>)/1048576 80328 ns 80468 ns 1.0x rng::fill_n(vector<int>)/32 1.99 ns 1.35 ns 1.5x rng::fill_n(vector<int>)/50 2.66 ns 2.12 ns 1.3x rng::fill_n(vector<int>)/1024 37.7 ns 35.8 ns 1.1x rng::fill_n(vector<int>)/8192 253 ns 250 ns 1.0x rng::fill_n(vector<int>)/65536 2922 ns 2930 ns 1.0x rng::fill_n(vector<int>)/1048576 79739 ns 79742 ns 1.0x ``` #### `fill` ``` -------------------------------------------------------------------------- Benchmark Before After Speedup -------------------------------------------------------------------------- std::fill(deque<int>)/32 13.7 ns 2.45 ns 5.6x std::fill(deque<int>)/50 21.7 ns 4.57 ns 4.7x std::fill(deque<int>)/1024 367 ns 38.5 ns 9.5x std::fill(deque<int>)/8192 2896 ns 247 ns 11.7x std::fill(deque<int>)/65536 23723 ns 2907 ns 8.2x std::fill(deque<int>)/1048576 379043 ns 79885 ns 4.7x rng::fill(deque<int>)/32 13.6 ns 2.70 ns 5.0x rng::fill(deque<int>)/50 23.4 ns 3.94 ns 5.9x rng::fill(deque<int>)/1024 377 ns 37.9 ns 9.9x rng::fill(deque<int>)/8192 2914 ns 286 ns 10.2x rng::fill(deque<int>)/65536 23612 ns 2939 ns 8.0x rng::fill(deque<int>)/1048576 379841 ns 80079 ns 4.7x std::fill(vector<int>)/32 1.99 ns 1.79 ns 1.1x std::fill(vector<int>)/50 3.05 ns 3.06 ns 1.0x std::fill(vector<int>)/1024 37.6 ns 38.0 ns 1.0x std::fill(vector<int>)/8192 255 ns 257 ns 1.0x std::fill(vector<int>)/65536 2966 ns 2981 ns 1.0x std::fill(vector<int>)/1048576 78300 ns 80348 ns 1.0x rng::fill(vector<int>)/32 1.77 ns 1.75 ns 1.0x rng::fill(vector<int>)/50 4.85 ns 2.31 ns 2.1x rng::fill(vector<int>)/1024 39.6 ns 36.1 ns 1.1x rng::fill(vector<int>)/8192 238 ns 251 ns 0.9x rng::fill(vector<int>)/65536 2941 ns 2918 ns 1.0x rng::fill(vector<int>)/1048576 80497 ns 80442 ns 1.0x ``` --------- Co-authored-by: Louis Dionne <ldionne.2@gmail.com> Co-authored-by: A. Jiang <de34@live.cn>	2025-10-17 07:41:24 +08:00
A. Jiang	d6d5026b51	[libc++] Avoid transitive inclusion for <__algorithm/find.h> (#162508) ... Currently, `size_t` and `__libcpp_is_constant_evaluated` are obtained by transitive inclusion in `<__algorithm/find.h>` when `<cwchar>` is not included. This broke module build when `_LIBCPP_HAS_WIDE_CHARACTERS` is `1` and caused CI failure. We should explicitly include the corresponding internal headers.	2025-10-09 17:57:59 +08:00
Nikolas Klauser	70b7874ab2	[libc++] Make the naming of the iterator_traits aliases consistent (#161661) ... This renames all the `iterator_traits` alises to be `__iterator_<type-name>`. e.g `iterator_traits<T>::value_type` will be `__iterator_value_type<T>`.	2025-10-07 14:42:11 +02:00
Nikolas Klauser	55803b8af1	Reapply "[libc++] Avoid constructing additional objects when using map::at" (#160738) (#161485) ... This reverts commit b86aaacf28b358b187071bc87075f1faa2d65c4e. The issue in LLVM has been fixed now.	2025-10-02 11:15:33 +02:00
Nikolas Klauser	66558d70dc	[libc++] Fix <__algorithm/find.h> when using -flax-vector-conversions=none (#161362)	2025-10-02 10:05:26 +02:00
Nikolas Klauser	97367d1046	[libc++] Vectorize std::find (#156431) ... ``` Apple M4: ----------------------------------------------------------------------------- Benchmark old new ----------------------------------------------------------------------------- std::find(vector<char>) (bail 25%)/8 1.43 ns 1.44 ns std::find(vector<char>) (bail 25%)/1024 5.54 ns 5.59 ns std::find(vector<char>) (bail 25%)/8192 38.4 ns 39.1 ns std::find(vector<char>) (bail 25%)/32768 134 ns 136 ns std::find(vector<int>) (bail 25%)/8 1.56 ns 1.57 ns std::find(vector<int>) (bail 25%)/1024 65.3 ns 65.4 ns std::find(vector<int>) (bail 25%)/8192 465 ns 464 ns std::find(vector<int>) (bail 25%)/32768 1832 ns 1832 ns std::find(vector<long long>) (bail 25%)/8 0.920 ns 1.20 ns std::find(vector<long long>) (bail 25%)/1024 65.2 ns 31.2 ns std::find(vector<long long>) (bail 25%)/8192 464 ns 255 ns std::find(vector<long long>) (bail 25%)/32768 1833 ns 992 ns std::find(vector<char>) (process all)/8 1.21 ns 1.22 ns std::find(vector<char>) (process all)/50 1.92 ns 1.93 ns std::find(vector<char>) (process all)/1024 16.6 ns 16.9 ns std::find(vector<char>) (process all)/8192 134 ns 136 ns std::find(vector<char>) (process all)/32768 488 ns 503 ns std::find(vector<int>) (process all)/8 2.45 ns 2.48 ns std::find(vector<int>) (process all)/50 12.7 ns 12.7 ns std::find(vector<int>) (process all)/1024 236 ns 236 ns std::find(vector<int>) (process all)/8192 1830 ns 1834 ns std::find(vector<int>) (process all)/32768 7351 ns 7346 ns std::find(vector<long long>) (process all)/8 2.02 ns 1.45 ns std::find(vector<long long>) (process all)/50 12.0 ns 6.12 ns std::find(vector<long long>) (process all)/1024 235 ns 123 ns std::find(vector<long long>) (process all)/8192 1830 ns 983 ns std::find(vector<long long>) (process all)/32768 7306 ns 3969 ns std::find(vector<bool>) (process all)/8 1.14 ns 1.15 ns std::find(vector<bool>) (process all)/50 1.16 ns 1.17 ns std::find(vector<bool>) (process all)/1024 4.51 ns 4.53 ns std::find(vector<bool>) (process all)/8192 33.6 ns 33.5 ns std::find(vector<bool>) (process all)/1048576 3660 ns 3660 ns ```	2025-09-29 11:10:19 +02:00
Andrew Lazarev	b86aaacf28	Revert "[libc++] Avoid constructing additional objects when using map::at" (#160738) ... Reverts llvm/llvm-project#157866 It breaks a lot of sanitizer buildbots	2025-09-25 15:10:37 -07:00
Nikolas Klauser	833d5f0cd8	[libc++] Avoid constructing additional objects when using map::at (#157866) ... This patch adds additional overloads to `map::at` in case its known that the argument is transparently comparable to the key type. This avoids actually constructing the key type in some cases, potentially removing allocations. ``` -------------------------------------------------------- Benchmark old new -------------------------------------------------------- BM_map_find_string_literal 12.8 ns 2.68 ns ```	2025-09-25 11:35:41 +02:00
Prabhu Rajasekaran	59b4621b6a	[libc++] Fix use of static in constexpr (#160180) ... `static` is not allowed inside constexpr functions in C++ versions below 23. This is causing a build error when compiled with GCC and warning when compiled with Clang. This patch fixes this. --------- Co-authored-by: A. Jiang <de34@live.cn>	2025-09-24 07:26:14 +08:00
Nikolas Klauser	04518e7357	[libc++] Improve the performance of std::make_heap a bit (#154092) ... ``` Apple M4: ----------------------------------------------------------------------------------------------------- Benchmark old new ----------------------------------------------------------------------------------------------------- BM_MakeHeap_uint32_Random_1 0.285 ns 0.271 ns BM_MakeHeap_uint32_Random_4 2.09 ns 1.80 ns BM_MakeHeap_uint32_Random_16 1.85 ns 1.83 ns BM_MakeHeap_uint32_Random_64 1.92 ns 1.50 ns BM_MakeHeap_uint32_Random_256 2.10 ns 1.87 ns BM_MakeHeap_uint32_Random_1024 1.73 ns 1.86 ns BM_MakeHeap_uint32_Random_16384 2.17 ns 2.05 ns BM_MakeHeap_uint32_Random_262144 1.77 ns 1.77 ns BM_MakeHeap_uint32_Ascending_1 0.288 ns 0.277 ns BM_MakeHeap_uint32_Ascending_4 0.658 ns 0.481 ns BM_MakeHeap_uint32_Ascending_16 0.636 ns 0.637 ns BM_MakeHeap_uint32_Ascending_64 0.643 ns 0.601 ns BM_MakeHeap_uint32_Ascending_256 0.710 ns 0.636 ns BM_MakeHeap_uint32_Ascending_1024 0.747 ns 0.660 ns BM_MakeHeap_uint32_Ascending_16384 0.713 ns 0.633 ns BM_MakeHeap_uint32_Ascending_262144 0.769 ns 0.731 ns BM_MakeHeap_uint32_Descending_1 0.294 ns 0.280 ns BM_MakeHeap_uint32_Descending_4 0.379 ns 0.305 ns BM_MakeHeap_uint32_Descending_16 0.376 ns 0.268 ns BM_MakeHeap_uint32_Descending_64 0.358 ns 0.271 ns BM_MakeHeap_uint32_Descending_256 0.377 ns 0.284 ns BM_MakeHeap_uint32_Descending_1024 0.355 ns 0.267 ns BM_MakeHeap_uint32_Descending_16384 0.348 ns 0.248 ns BM_MakeHeap_uint32_Descending_262144 0.349 ns 0.247 ns BM_MakeHeap_uint32_SingleElement_1 0.292 ns 0.280 ns BM_MakeHeap_uint32_SingleElement_4 0.570 ns 0.332 ns BM_MakeHeap_uint32_SingleElement_16 0.635 ns 0.604 ns BM_MakeHeap_uint32_SingleElement_64 0.653 ns 0.567 ns BM_MakeHeap_uint32_SingleElement_256 0.703 ns 0.609 ns BM_MakeHeap_uint32_SingleElement_1024 0.737 ns 0.604 ns BM_MakeHeap_uint32_SingleElement_16384 0.699 ns 0.574 ns BM_MakeHeap_uint32_SingleElement_262144 0.803 ns 0.684 ns BM_MakeHeap_uint32_PipeOrgan_1 0.291 ns 0.284 ns BM_MakeHeap_uint32_PipeOrgan_4 0.588 ns 0.399 ns BM_MakeHeap_uint32_PipeOrgan_16 0.648 ns 1.12 ns BM_MakeHeap_uint32_PipeOrgan_64 0.662 ns 0.771 ns BM_MakeHeap_uint32_PipeOrgan_256 0.723 ns 0.672 ns BM_MakeHeap_uint32_PipeOrgan_1024 0.749 ns 0.674 ns BM_MakeHeap_uint32_PipeOrgan_16384 0.708 ns 0.638 ns BM_MakeHeap_uint32_PipeOrgan_262144 0.786 ns 0.743 ns BM_MakeHeap_uint32_Heap_1 0.298 ns 0.282 ns BM_MakeHeap_uint32_Heap_4 0.396 ns 0.308 ns BM_MakeHeap_uint32_Heap_16 0.377 ns 0.268 ns BM_MakeHeap_uint32_Heap_64 0.356 ns 0.271 ns BM_MakeHeap_uint32_Heap_256 0.378 ns 0.290 ns BM_MakeHeap_uint32_Heap_1024 0.356 ns 0.275 ns BM_MakeHeap_uint32_Heap_16384 0.348 ns 0.252 ns BM_MakeHeap_uint32_Heap_262144 0.347 ns 0.250 ns BM_MakeHeap_uint32_QuickSortAdversary_1 0.290 ns 0.284 ns BM_MakeHeap_uint32_QuickSortAdversary_4 0.627 ns 0.409 ns BM_MakeHeap_uint32_QuickSortAdversary_16 0.640 ns 0.653 ns BM_MakeHeap_uint32_QuickSortAdversary_64 0.577 ns 0.484 ns BM_MakeHeap_uint32_QuickSortAdversary_256 0.613 ns 0.521 ns BM_MakeHeap_uint32_QuickSortAdversary_1024 0.652 ns 0.514 ns BM_MakeHeap_uint32_QuickSortAdversary_16384 0.428 ns 0.308 ns BM_MakeHeap_uint32_QuickSortAdversary_262144 0.373 ns 0.261 ns BM_MakeHeap_uint64_Random_1 0.291 ns 0.281 ns BM_MakeHeap_uint64_Random_4 2.20 ns 1.97 ns BM_MakeHeap_uint64_Random_16 1.93 ns 1.70 ns BM_MakeHeap_uint64_Random_64 1.89 ns 1.48 ns BM_MakeHeap_uint64_Random_256 2.10 ns 1.99 ns BM_MakeHeap_uint64_Random_1024 1.41 ns 2.04 ns BM_MakeHeap_uint64_Random_16384 2.12 ns 1.83 ns BM_MakeHeap_uint64_Random_262144 1.83 ns 1.63 ns BM_MakeHeap_uint64_Ascending_1 0.288 ns 0.283 ns BM_MakeHeap_uint64_Ascending_4 0.624 ns 0.493 ns BM_MakeHeap_uint64_Ascending_16 0.648 ns 0.688 ns BM_MakeHeap_uint64_Ascending_64 0.671 ns 0.634 ns BM_MakeHeap_uint64_Ascending_256 0.739 ns 0.680 ns BM_MakeHeap_uint64_Ascending_1024 0.761 ns 0.698 ns BM_MakeHeap_uint64_Ascending_16384 0.740 ns 0.685 ns BM_MakeHeap_uint64_Ascending_262144 0.849 ns 0.837 ns BM_MakeHeap_uint64_Descending_1 0.304 ns 0.287 ns BM_MakeHeap_uint64_Descending_4 0.395 ns 0.312 ns BM_MakeHeap_uint64_Descending_16 0.374 ns 0.276 ns BM_MakeHeap_uint64_Descending_64 0.358 ns 0.272 ns BM_MakeHeap_uint64_Descending_256 0.389 ns 0.303 ns BM_MakeHeap_uint64_Descending_1024 0.361 ns 0.278 ns BM_MakeHeap_uint64_Descending_16384 0.352 ns 0.253 ns BM_MakeHeap_uint64_Descending_262144 0.350 ns 0.251 ns BM_MakeHeap_uint64_SingleElement_1 0.295 ns 0.285 ns BM_MakeHeap_uint64_SingleElement_4 0.569 ns 0.358 ns BM_MakeHeap_uint64_SingleElement_16 0.649 ns 0.652 ns BM_MakeHeap_uint64_SingleElement_64 0.673 ns 0.565 ns BM_MakeHeap_uint64_SingleElement_256 0.732 ns 0.651 ns BM_MakeHeap_uint64_SingleElement_1024 0.759 ns 0.632 ns BM_MakeHeap_uint64_SingleElement_16384 0.748 ns 0.614 ns BM_MakeHeap_uint64_SingleElement_262144 0.947 ns 0.797 ns BM_MakeHeap_uint64_PipeOrgan_1 0.295 ns 0.284 ns BM_MakeHeap_uint64_PipeOrgan_4 0.601 ns 0.496 ns BM_MakeHeap_uint64_PipeOrgan_16 0.655 ns 1.18 ns BM_MakeHeap_uint64_PipeOrgan_64 0.682 ns 0.803 ns BM_MakeHeap_uint64_PipeOrgan_256 0.759 ns 0.710 ns BM_MakeHeap_uint64_PipeOrgan_1024 0.759 ns 0.713 ns BM_MakeHeap_uint64_PipeOrgan_16384 0.739 ns 0.696 ns BM_MakeHeap_uint64_PipeOrgan_262144 0.870 ns 0.849 ns BM_MakeHeap_uint64_Heap_1 0.290 ns 0.284 ns BM_MakeHeap_uint64_Heap_4 0.413 ns 0.314 ns BM_MakeHeap_uint64_Heap_16 0.378 ns 0.277 ns BM_MakeHeap_uint64_Heap_64 0.362 ns 0.272 ns BM_MakeHeap_uint64_Heap_256 0.389 ns 0.303 ns BM_MakeHeap_uint64_Heap_1024 0.362 ns 0.283 ns BM_MakeHeap_uint64_Heap_16384 0.352 ns 0.253 ns BM_MakeHeap_uint64_Heap_262144 0.350 ns 0.251 ns BM_MakeHeap_uint64_QuickSortAdversary_1 0.293 ns 0.284 ns BM_MakeHeap_uint64_QuickSortAdversary_4 0.606 ns 0.494 ns BM_MakeHeap_uint64_QuickSortAdversary_16 0.645 ns 0.691 ns BM_MakeHeap_uint64_QuickSortAdversary_64 0.607 ns 0.511 ns BM_MakeHeap_uint64_QuickSortAdversary_256 0.641 ns 0.537 ns BM_MakeHeap_uint64_QuickSortAdversary_1024 0.657 ns 0.529 ns BM_MakeHeap_uint64_QuickSortAdversary_16384 0.435 ns 0.316 ns BM_MakeHeap_uint64_QuickSortAdversary_262144 0.382 ns 0.266 ns BM_MakeHeap_pair<uint32, uint32>_Random_1 0.297 ns 0.378 ns BM_MakeHeap_pair<uint32, uint32>_Random_4 2.80 ns 0.765 ns BM_MakeHeap_pair<uint32, uint32>_Random_16 2.92 ns 2.20 ns BM_MakeHeap_pair<uint32, uint32>_Random_64 3.17 ns 3.64 ns BM_MakeHeap_pair<uint32, uint32>_Random_256 3.44 ns 3.20 ns BM_MakeHeap_pair<uint32, uint32>_Random_1024 3.35 ns 3.64 ns BM_MakeHeap_pair<uint32, uint32>_Random_16384 3.22 ns 3.50 ns BM_MakeHeap_pair<uint32, uint32>_Random_262144 3.61 ns 3.46 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_1 0.291 ns 0.379 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_4 0.779 ns 0.436 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_16 0.943 ns 1.01 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_64 1.17 ns 1.26 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_256 1.38 ns 1.44 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_1024 1.37 ns 1.43 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_16384 1.29 ns 1.31 ns BM_MakeHeap_pair<uint32, uint32>_Ascending_262144 1.37 ns 1.40 ns BM_MakeHeap_pair<uint32, uint32>_Descending_1 0.292 ns 0.382 ns BM_MakeHeap_pair<uint32, uint32>_Descending_4 0.440 ns 0.347 ns BM_MakeHeap_pair<uint32, uint32>_Descending_16 0.529 ns 0.520 ns BM_MakeHeap_pair<uint32, uint32>_Descending_64 0.540 ns 0.527 ns BM_MakeHeap_pair<uint32, uint32>_Descending_256 0.637 ns 0.583 ns BM_MakeHeap_pair<uint32, uint32>_Descending_1024 0.552 ns 0.513 ns BM_MakeHeap_pair<uint32, uint32>_Descending_16384 0.522 ns 0.488 ns BM_MakeHeap_pair<uint32, uint32>_Descending_262144 0.515 ns 0.492 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_1 0.299 ns 0.377 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_4 0.787 ns 0.474 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_16 1.07 ns 0.921 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_64 1.20 ns 1.15 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_256 1.30 ns 1.27 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_1024 1.30 ns 1.31 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_16384 1.29 ns 1.28 ns BM_MakeHeap_pair<uint32, uint32>_SingleElement_262144 1.37 ns 1.38 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_1 0.293 ns 0.385 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_4 0.677 ns 0.438 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_16 1.04 ns 1.00 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_64 1.20 ns 1.27 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_256 1.40 ns 1.43 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_1024 1.36 ns 1.43 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_16384 1.27 ns 1.31 ns BM_MakeHeap_pair<uint32, uint32>_PipeOrgan_262144 1.37 ns 1.41 ns BM_MakeHeap_pair<uint32, uint32>_Heap_1 0.292 ns 0.378 ns BM_MakeHeap_pair<uint32, uint32>_Heap_4 0.440 ns 0.380 ns BM_MakeHeap_pair<uint32, uint32>_Heap_16 0.560 ns 0.606 ns BM_MakeHeap_pair<uint32, uint32>_Heap_64 0.588 ns 0.573 ns BM_MakeHeap_pair<uint32, uint32>_Heap_256 0.632 ns 0.607 ns BM_MakeHeap_pair<uint32, uint32>_Heap_1024 0.598 ns 0.580 ns BM_MakeHeap_pair<uint32, uint32>_Heap_16384 0.576 ns 0.563 ns BM_MakeHeap_pair<uint32, uint32>_Heap_262144 0.572 ns 0.561 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_1 0.304 ns 0.379 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_4 0.823 ns 0.430 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_16 1.08 ns 1.03 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_64 1.18 ns 1.23 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_256 1.39 ns 1.43 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_1024 1.36 ns 1.42 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_16384 1.28 ns 1.32 ns BM_MakeHeap_pair<uint32, uint32>_QuickSortAdversary_262144 1.34 ns 1.37 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_1 0.276 ns 0.511 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_4 4.25 ns 1.96 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_16 4.84 ns 3.77 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_64 5.53 ns 4.93 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_256 5.30 ns 5.06 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_1024 5.29 ns 5.02 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_16384 5.53 ns 5.31 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Random_262144 5.49 ns 5.29 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_1 0.275 ns 0.443 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_4 1.22 ns 0.764 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_16 1.39 ns 1.49 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_64 1.66 ns 1.76 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_256 1.85 ns 1.99 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_1024 1.90 ns 2.02 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_16384 2.15 ns 2.27 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Ascending_262144 2.25 ns 2.37 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_1 0.274 ns 0.445 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_4 0.939 ns 0.520 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_16 1.02 ns 0.811 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_64 1.01 ns 0.941 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_256 1.03 ns 1.01 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_1024 0.969 ns 0.947 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_16384 0.882 ns 0.876 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Descending_262144 0.893 ns 0.871 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_1 0.276 ns 0.443 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_4 1.34 ns 0.870 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_16 1.60 ns 1.59 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_64 1.91 ns 2.00 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_256 1.91 ns 2.08 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_1024 1.91 ns 2.10 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_16384 2.13 ns 2.35 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_SingleElement_262144 2.25 ns 2.48 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_1 0.275 ns 0.446 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_4 1.07 ns 0.671 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_16 1.36 ns 1.44 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_64 1.70 ns 1.80 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_256 1.88 ns 2.05 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_1024 1.92 ns 2.06 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_16384 2.11 ns 2.26 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_PipeOrgan_262144 2.18 ns 2.34 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_1 0.274 ns 0.441 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_4 0.938 ns 0.587 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_16 1.01 ns 0.873 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_64 1.08 ns 1.00 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_256 1.21 ns 1.18 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_1024 1.37 ns 1.29 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_16384 1.31 ns 1.26 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_Heap_262144 1.29 ns 1.25 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_1 0.275 ns 0.447 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_4 1.22 ns 0.764 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_16 1.35 ns 1.46 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_64 1.63 ns 1.73 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_256 1.83 ns 1.87 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_1024 1.81 ns 1.94 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_16384 2.06 ns 2.19 ns BM_MakeHeap_tuple<uint32, uint64, uint32>_QuickSortAdversary_262144 2.14 ns 2.31 ns BM_MakeHeap_string_Random_1 0.289 ns 0.446 ns BM_MakeHeap_string_Random_4 7.43 ns 4.84 ns BM_MakeHeap_string_Random_16 9.73 ns 8.92 ns BM_MakeHeap_string_Random_64 11.5 ns 11.5 ns BM_MakeHeap_string_Random_256 12.6 ns 12.4 ns BM_MakeHeap_string_Random_1024 13.2 ns 13.2 ns BM_MakeHeap_string_Random_16384 15.4 ns 15.4 ns BM_MakeHeap_string_Random_262144 21.4 ns 21.5 ns BM_MakeHeap_string_Ascending_1 0.287 ns 0.447 ns BM_MakeHeap_string_Ascending_4 3.22 ns 2.44 ns BM_MakeHeap_string_Ascending_16 3.77 ns 3.62 ns BM_MakeHeap_string_Ascending_64 4.84 ns 5.17 ns BM_MakeHeap_string_Ascending_256 5.79 ns 6.04 ns BM_MakeHeap_string_Ascending_1024 5.93 ns 6.60 ns BM_MakeHeap_string_Ascending_16384 6.84 ns 7.25 ns BM_MakeHeap_string_Ascending_262144 13.5 ns 14.3 ns BM_MakeHeap_string_Descending_1 0.293 ns 0.447 ns BM_MakeHeap_string_Descending_4 2.61 ns 1.83 ns BM_MakeHeap_string_Descending_16 2.64 ns 2.60 ns BM_MakeHeap_string_Descending_64 2.75 ns 2.74 ns BM_MakeHeap_string_Descending_256 3.78 ns 3.57 ns BM_MakeHeap_string_Descending_1024 3.20 ns 3.51 ns BM_MakeHeap_string_Descending_16384 3.57 ns 3.85 ns BM_MakeHeap_string_Descending_262144 6.27 ns 6.39 ns BM_MakeHeap_string_SingleElement_1 0.291 ns 0.448 ns BM_MakeHeap_string_SingleElement_4 3.88 ns 2.71 ns BM_MakeHeap_string_SingleElement_16 5.08 ns 4.96 ns BM_MakeHeap_string_SingleElement_64 6.14 ns 6.29 ns BM_MakeHeap_string_SingleElement_256 6.13 ns 6.46 ns BM_MakeHeap_string_SingleElement_1024 5.98 ns 6.60 ns BM_MakeHeap_string_SingleElement_16384 5.88 ns 6.39 ns BM_MakeHeap_string_SingleElement_262144 9.95 ns 10.2 ns BM_MakeHeap_string_PipeOrgan_1 0.292 ns 0.447 ns BM_MakeHeap_string_PipeOrgan_4 2.97 ns 2.51 ns BM_MakeHeap_string_PipeOrgan_16 3.76 ns 3.91 ns BM_MakeHeap_string_PipeOrgan_64 4.89 ns 5.20 ns BM_MakeHeap_string_PipeOrgan_256 5.77 ns 6.09 ns BM_MakeHeap_string_PipeOrgan_1024 6.14 ns 6.40 ns BM_MakeHeap_string_PipeOrgan_16384 6.83 ns 7.32 ns BM_MakeHeap_string_PipeOrgan_262144 13.8 ns 14.6 ns BM_MakeHeap_string_Heap_1 0.288 ns 0.515 ns BM_MakeHeap_string_Heap_4 3.62 ns 4.20 ns BM_MakeHeap_string_Heap_16 5.36 ns 5.23 ns BM_MakeHeap_string_Heap_64 5.79 ns 5.38 ns BM_MakeHeap_string_Heap_256 5.70 ns 5.40 ns BM_MakeHeap_string_Heap_1024 5.78 ns 5.37 ns BM_MakeHeap_string_Heap_16384 6.09 ns 5.67 ns BM_MakeHeap_string_Heap_262144 6.37 ns 5.96 ns BM_MakeHeap_string_QuickSortAdversary_1 0.282 ns 0.448 ns BM_MakeHeap_string_QuickSortAdversary_4 7.45 ns 5.60 ns BM_MakeHeap_string_QuickSortAdversary_16 9.76 ns 8.85 ns BM_MakeHeap_string_QuickSortAdversary_64 11.5 ns 11.2 ns BM_MakeHeap_string_QuickSortAdversary_256 12.0 ns 11.8 ns BM_MakeHeap_string_QuickSortAdversary_1024 12.2 ns 12.0 ns BM_MakeHeap_string_QuickSortAdversary_16384 13.7 ns 13.6 ns BM_MakeHeap_string_QuickSortAdversary_262144 14.1 ns 14.8 ns BM_MakeHeap_float_Random_1 0.287 ns 0.287 ns BM_MakeHeap_float_Random_4 2.29 ns 2.60 ns BM_MakeHeap_float_Random_16 4.00 ns 2.48 ns BM_MakeHeap_float_Random_64 4.41 ns 1.92 ns BM_MakeHeap_float_Random_256 4.73 ns 2.05 ns BM_MakeHeap_float_Random_1024 4.90 ns 2.27 ns BM_MakeHeap_float_Random_16384 4.42 ns 2.27 ns BM_MakeHeap_float_Random_262144 4.72 ns 1.39 ns BM_MakeHeap_float_Ascending_1 0.291 ns 0.293 ns BM_MakeHeap_float_Ascending_4 0.633 ns 0.428 ns BM_MakeHeap_float_Ascending_16 0.638 ns 0.874 ns BM_MakeHeap_float_Ascending_64 0.614 ns 0.698 ns BM_MakeHeap_float_Ascending_256 0.663 ns 0.713 ns BM_MakeHeap_float_Ascending_1024 0.660 ns 0.761 ns BM_MakeHeap_float_Ascending_16384 0.628 ns 0.725 ns BM_MakeHeap_float_Ascending_262144 0.629 ns 0.814 ns BM_MakeHeap_float_Descending_1 0.290 ns 0.290 ns BM_MakeHeap_float_Descending_4 0.421 ns 0.316 ns BM_MakeHeap_float_Descending_16 0.302 ns 0.225 ns BM_MakeHeap_float_Descending_64 0.293 ns 0.212 ns BM_MakeHeap_float_Descending_256 0.314 ns 0.246 ns BM_MakeHeap_float_Descending_1024 0.300 ns 0.231 ns BM_MakeHeap_float_Descending_16384 0.308 ns 0.205 ns BM_MakeHeap_float_Descending_262144 0.309 ns 0.203 ns BM_MakeHeap_float_SingleElement_1 0.289 ns 0.292 ns BM_MakeHeap_float_SingleElement_4 0.569 ns 0.347 ns BM_MakeHeap_float_SingleElement_16 0.538 ns 0.825 ns BM_MakeHeap_float_SingleElement_64 0.585 ns 0.727 ns BM_MakeHeap_float_SingleElement_256 0.603 ns 0.708 ns BM_MakeHeap_float_SingleElement_1024 0.618 ns 0.760 ns BM_MakeHeap_float_SingleElement_16384 0.599 ns 0.726 ns BM_MakeHeap_float_SingleElement_262144 0.723 ns 0.820 ns BM_MakeHeap_float_PipeOrgan_1 0.289 ns 0.291 ns BM_MakeHeap_float_PipeOrgan_4 0.457 ns 0.420 ns BM_MakeHeap_float_PipeOrgan_16 0.670 ns 1.32 ns BM_MakeHeap_float_PipeOrgan_64 0.764 ns 0.889 ns BM_MakeHeap_float_PipeOrgan_256 0.793 ns 0.757 ns BM_MakeHeap_float_PipeOrgan_1024 0.755 ns 0.764 ns BM_MakeHeap_float_PipeOrgan_16384 0.723 ns 0.723 ns BM_MakeHeap_float_PipeOrgan_262144 0.654 ns 0.817 ns BM_MakeHeap_float_Heap_1 0.291 ns 0.289 ns BM_MakeHeap_float_Heap_4 0.388 ns 0.316 ns BM_MakeHeap_float_Heap_16 0.317 ns 0.225 ns BM_MakeHeap_float_Heap_64 0.353 ns 0.213 ns BM_MakeHeap_float_Heap_256 0.361 ns 0.246 ns BM_MakeHeap_float_Heap_1024 0.381 ns 0.233 ns BM_MakeHeap_float_Heap_16384 0.390 ns 0.205 ns BM_MakeHeap_float_Heap_262144 0.379 ns 0.202 ns BM_MakeHeap_float_QuickSortAdversary_1 0.295 ns 0.289 ns BM_MakeHeap_float_QuickSortAdversary_4 0.640 ns 0.422 ns BM_MakeHeap_float_QuickSortAdversary_16 0.658 ns 0.871 ns BM_MakeHeap_float_QuickSortAdversary_64 0.574 ns 0.659 ns BM_MakeHeap_float_QuickSortAdversary_256 0.631 ns 0.550 ns BM_MakeHeap_float_QuickSortAdversary_1024 0.617 ns 0.552 ns BM_MakeHeap_float_QuickSortAdversary_16384 0.424 ns 0.283 ns BM_MakeHeap_float_QuickSortAdversary_262144 0.386 ns 0.219 ns ``` Fixes #120752	2025-09-08 12:00:04 +02:00
Nikolas Klauser	34d4f0c136	[libc++][NFC] Use llvm.org/PR to link to bug reports (#156288) ... We've built up quite a few links directly to github within the code base. We should instead use `llvm.org/PR<issue-number>` to link to bugs, since that is resilient to the bug tracker changing in the future. This is especially relevant for tests linking to bugs, since they will probably be there for decades to come. A nice side effect is that these links are significantly shorter than the GH links, making them much less of an eyesore. This patch also replaces a few links that linked to the old bugzilla instance on llvm.org.	2025-09-04 09:20:02 +02:00
Louis Dionne	2ae4b92a1c	[libc++] Fix broken precondition of __bit_log2 (#155476) ... In #135303, we started using `__bit_log2` instead of `__log2i` inside `std::sort`. However, `__bit_log2` has a precondition that `__log2i` didn't have, which is that the input is non-zero. While it technically makes no sense to request the logarithm of 0, `__log2i` handled that case and returned 0 without issues. After switching to `__bit_log2`, passing 0 as an input results in an unsigned integer overflow which can trigger `-fsanitize=unsigned-integer-overflow`. While not technically UB in itself, it's clearly not intended either. To fix this, we add an internal assertion to `__bit_log2` which catches the issue in our test suite, and we make sure not to violate `__bit_log2`'s preconditions before we call it from `std::sort`.	2025-08-28 18:07:59 -04:00
Nikolas Klauser	cdb67e1131	[libc++][NFC] Make __is_segmented_iterator a variable template (#149976)	2025-07-25 16:27:34 +02:00
Louis Dionne	d145f716ae	[libc++] Bump Xcode support (#148651) ... Libc++'s policy is to support only the latest released Xcode, which is Xcode 16.x. We did update our CI jobs to Xcode 16.x, but we forgot to update the documentation, which still mentioned Xcode 15. This patch updates the documentation and cleans up outdated mentions of apple-clang-15 in the test suite.	2025-07-15 10:45:37 -04:00
Peng Liu	9827440f1e	[libc++] Optimize ranges::{for_each, for_each_n} for segmented iterators (#132896) ... Previously, the segmented iterator optimization was limited to `std::{for_each, for_each_n}`. This patch extends the optimization to `std::ranges::for_each` and `std::ranges::for_each_n`, ensuring consistent optimizations across these algorithms. This patch first generalizes the `std` algorithms by introducing a `Projection` parameter, which is set to `__identity` for the `std` algorithms. Then we let the `ranges` algorithms to directly call their `std` counterparts with a general `__proj` argument. Benchmarks demonstrate performance improvements of up to 21.4x for ``std::deque::iterator`` and 22.3x for ``join_view`` of ``vector<vector<char>>``. Addresses a subtask of #102817.	2025-06-18 12:22:47 -04:00
James Y Knight	c3656afdf1	Revert "[libc++] Introduce ABI sensitive areas to avoid requiring _LIBCPP_HIDE_FROM_ABI everywhere (#131156)" (#141756) ... This reverts commit c861fe8a71e64f3d2108c58147e7375cd9314521. Unfortunately, this use of hidden visibility attributes causes user-defined specializations of standard-library types to also be marked hidden by default, which is incorrect. See discussion thread on #131156. ...and also reverts the follow-up commits: Revert "[libc++] Add explicit ABI annotations to functions from the block runtime declared in <__functional/function.h> (#140592)" This reverts commit 3e4c9dc299c35155934688184319d391b298fff7. Revert "[libc++] Make ABI annotations explicit for windows-specific code (#140507)" This reverts commit f73287e623a6c2e4a3485832bc3e10860cd26eb5. Revert "[libc++][NFC] Replace a few "namespace std" with the correct macro (#140510)" This reverts commit 1d411f27c769a32cb22ce50b9dc4421e34fd40dd.	2025-05-28 12:04:51 -04:00
Peng Liu	09c266b75d	[libc++] Optimize std::for_each_n for segmented iterators (#135468) ... This patch enhances the performance of `std::for_each_n` when used with segmented iterators, leading to significant performance improvements, summarized in the tables below. This addresses a subtask of https://github.com/llvm/llvm-project/issues/102817.	2025-05-21 12:10:50 -04:00
Nikolas Klauser	c861fe8a71	[libc++] Introduce ABI sensitive areas to avoid requiring _LIBCPP_HIDE_FROM_ABI everywhere (#131156) ... This patch introduces `_LIBCPP_{BEGIN,END}_EXPLICIT_ABI_ANNOTATIONS`, which allow us to have implicit annotations for most functions, and just where it's not "hide_from_abi everything" we add explicit annotations. This allows us to drop the `_LIBCPP_HIDE_FROM_ABI` macro from most functions in libc++.	2025-05-18 15:47:05 +02:00
Peng Liu	e9280a1d39	[libc++] Backport segmented iterator optimization for std::for_each to C++11 (#134960) ... Previously, the segmented iterator optimization for `std::for_each` was restricted to C++23 and later due to its dependency on `__movable_box`, which is not available in earlier standards. This patch eliminates that restriction, enabling consistent optimizations starting from C++11. By backporting this enhancement, we improve performance across older standards and create opportunities to extend similar optimizations to other algorithms by forwarding their calls to `std::for_each`.	2025-04-19 07:12:43 -04:00
Peng Liu	9e3982d9ae	[libc++] Replace __libcpp_{ctz, clz} with __builtin_{ctzg, clzg} (#133920) ... `__libcpp_{ctz, clz}` were previously used as fallbacks for `__builtin_{ctzg, clzg}` to ensure compatibility with older compilers (Clang 18 and earlier), as `__builtin_{ctzg, clzg}` became available in Clang 19. Now that support for Clang 18 has been officially dropped in #130142, we can now safely replace all instances of `__libcpp_{ctz, clz}` with `__count{l,r}_zero` (which internally call `__builtin_{ctzg, clzg}` and eliminate the fallback logic. Closes #131179.	2025-04-18 20:57:05 -04:00
Дмитрий Изволов	51b8c66b08	[libc++] Extend the scope of radix sorting inside std::stable_sort to floating-point types (#129452) ... These changes speed up `std::stable_sort` in the case of sorting floating-point types. This applies only to IEEE 754 floats. The speedup is similar to that achieved for integers in PR #104683 (see benchmarks below). Why does this worth doing? Previously, `std::stable_sort` had almost no chance of beating `std::sort`. Now there are cases when `std::stable_sort` is preferrable, and the difference is significant. ``` --------------------------------------------------------------------------- Benchmark | std::stable_sort | std::sort | std::stable_sort | without radix_sort | | with radix_sort --------------------------------------------------------------------------- float_Random_1 | 1.62 ns | 2.15 ns | 1.61 ns float_Random_4 | 18.0 ns | 2.71 ns | 16.3 ns float_Random_16 | 118 ns | 113 ns | 112 ns float_Random_64 | 751 ns | 647 ns | 730 ns float_Random_256 | 4715 ns | 2937 ns | 4669 ns float_Random_1024 | 25713 ns | 13172 ns | 5959 ns <-- float_Random_4096 | 131307 ns | 56870 ns | 19294 ns <-- float_Random_16384 | 624996 ns | 242953 ns | 64264 ns <-- float_Random_65536 | 2895661 ns | 1027279 ns | 288553 ns <-- float_Random_262144 | 13285372 ns | 4342593 ns | 3022377 ns <-- float_Random_1048576 | 60595871 ns | 19087591 ns | 18690457 ns <-- float_Random_2097152 | 131336117 ns | 38800396 ns | 52325016 ns float_Random_4194304 | 270043042 ns | 79978019 ns | 102907726 ns double_Random_1 | 1.60 ns | 2.15 ns | 1.61 ns double_Random_4 | 15.2 ns | 2.70 ns | 16.9 ns double_Random_16 | 104 ns | 112 ns | 119 ns double_Random_64 | 712 ns | 614 ns | 755 ns double_Random_256 | 4496 ns | 2966 ns | 4820 ns double_Random_1024 | 24722 ns | 12679 ns | 6189 ns <-- double_Random_4096 | 126075 ns | 54484 ns | 20999 ns <-- double_Random_16384 | 613782 ns | 232557 ns | 110276 ns <-- double_Random_65536 | 2894972 ns | 988531 ns | 774302 ns <-- double_Random_262144 | 13460273 ns | 4278059 ns | 5115123 ns double_Random_1048576 | 61119996 ns | 18408462 ns | 27166574 ns double_Random_2097152 | 132511525 ns | 37986158 ns | 54423869 ns double_Random_4194304 | 272949862 ns | 77912616 ns | 147670834 ns ``` Comparison for only `std::stable_sort`: ``` Benchmark Time Time Old Time New -------------------------------------------------------------------------------------------------- BM_StableSort_float_Random_1024 -0.7997 25438 5096 BM_StableSort_float_Random_4096 -0.8731 128157 16260 BM_StableSort_float_Random_16384 -0.9024 621271 60623 BM_StableSort_float_Random_65536 -0.9081 2922413 268619 BM_StableSort_float_Random_262144 -0.7766 13386345 2990408 BM_StableSort_float_Random_1048576 -0.6954 60673010 18481751 BM_StableSort_float_Random_2097152 -0.6026 130977358 52052182 BM_StableSort_float_Random_4194304 -0.6252 271556583 101770500 BM_StableSort_float_Ascending_1024 -0.6430 6711 2396 BM_StableSort_float_Ascending_4096 -0.7979 38460 7773 BM_StableSort_float_Ascending_16384 -0.8471 191069 29222 BM_StableSort_float_Ascending_65536 -0.8683 882321 116194 BM_StableSort_float_Ascending_262144 -0.8346 3868552 639937 BM_StableSort_float_Ascending_1048576 -0.7460 16521233 4195953 BM_StableSort_float_Ascending_2097152 -0.5439 21757532 9922776 BM_StableSort_float_Ascending_4194304 -0.7525 67847496 16791582 BM_StableSort_float_Descending_1024 -0.6359 15038 5475 BM_StableSort_float_Descending_4096 -0.7090 62810 18278 BM_StableSort_float_Descending_16384 -0.7763 311844 69750 BM_StableSort_float_Descending_65536 -0.7228 1270513 352202 BM_StableSort_float_Descending_262144 -0.6785 5484173 1763045 BM_StableSort_float_Descending_1048576 -0.5084 20223149 9941852 BM_StableSort_float_Descending_2097152 -0.7646 60523254 14247014 BM_StableSort_float_Descending_4194304 -0.5638 95706839 41748858 BM_StableSort_float_SingleElement_1024 +0.3715 1732 2375 BM_StableSort_float_SingleElement_4096 -0.1685 9357 7781 BM_StableSort_float_SingleElement_16384 -0.3793 47307 29362 BM_StableSort_float_SingleElement_65536 -0.4925 227666 115536 BM_StableSort_float_SingleElement_262144 -0.4271 1075853 616387 BM_StableSort_float_SingleElement_1048576 -0.3736 5097599 3193279 BM_StableSort_float_SingleElement_2097152 -0.2470 9854161 7420158 BM_StableSort_float_SingleElement_4194304 -0.3384 22175964 14670720 BM_StableSort_float_PipeOrgan_1024 -0.4885 10664 5455 BM_StableSort_float_PipeOrgan_4096 -0.6340 50095 18337 BM_StableSort_float_PipeOrgan_16384 -0.7078 238700 69739 BM_StableSort_float_PipeOrgan_65536 -0.6740 1102419 359378 BM_StableSort_float_PipeOrgan_262144 -0.7460 4698739 1193511 BM_StableSort_float_PipeOrgan_1048576 -0.5657 18493972 8032392 BM_StableSort_float_PipeOrgan_2097152 -0.7116 41089206 11850349 BM_StableSort_float_PipeOrgan_4194304 -0.6650 83445011 27955737 BM_StableSort_float_QuickSortAdversary_1024 -0.6863 17402 5460 BM_StableSort_float_QuickSortAdversary_4096 -0.7715 79864 18247 BM_StableSort_float_QuickSortAdversary_16384 -0.7800 317480 69839 BM_StableSort_float_QuickSortAdversary_65536 -0.7400 1357601 352967 BM_StableSort_float_QuickSortAdversary_262144 -0.6450 5662094 2009769 BM_StableSort_float_QuickSortAdversary_1048576 -0.5092 21173627 10392107 BM_StableSort_float_QuickSortAdversary_2097152 -0.7333 61748178 16469993 BM_StableSort_float_QuickSortAdversary_4194304 -0.5607 98459863 43250182 BM_StableSort_double_Random_1024 -0.7657 24769 5802 BM_StableSort_double_Random_4096 -0.8441 126449 19717 BM_StableSort_double_Random_16384 -0.8269 614910 106447 BM_StableSort_double_Random_65536 -0.7413 2905000 751427 BM_StableSort_double_Random_262144 -0.6287 13449514 4994348 BM_StableSort_double_Random_1048576 -0.5635 60863246 26568349 BM_StableSort_double_Random_2097152 -0.5959 130293892 52654532 BM_StableSort_double_Random_4194304 -0.4772 272616445 142526267 BM_StableSort_double_Ascending_1024 -0.4870 6757 3466 BM_StableSort_double_Ascending_4096 -0.7360 37592 9923 BM_StableSort_double_Ascending_16384 -0.7971 183967 37324 BM_StableSort_double_Ascending_65536 -0.7465 897116 227398 BM_StableSort_double_Ascending_262144 -0.6764 4020980 1301033 BM_StableSort_double_Ascending_1048576 -0.6407 16421799 5900751 BM_StableSort_double_Ascending_2097152 -0.6380 29347139 10622419 BM_StableSort_double_Ascending_4194304 -0.5934 70439925 28644185 BM_StableSort_double_Descending_1024 -0.5988 15216 6105 BM_StableSort_double_Descending_4096 -0.6857 65069 20449 BM_StableSort_double_Descending_16384 -0.6922 329321 101381 BM_StableSort_double_Descending_65536 -0.7038 1367970 405242 BM_StableSort_double_Descending_262144 -0.6472 5361644 1891429 BM_StableSort_double_Descending_1048576 -0.6656 22031404 7366459 BM_StableSort_double_Descending_2097152 -0.7593 68922467 16591242 BM_StableSort_double_Descending_4194304 -0.6392 96283643 34743223 BM_StableSort_double_SingleElement_1024 +0.9128 1895 3625 BM_StableSort_double_SingleElement_4096 +0.1475 10013 11490 BM_StableSort_double_SingleElement_16384 -0.1901 52382 42424 BM_StableSort_double_SingleElement_65536 -0.2096 254698 201313 BM_StableSort_double_SingleElement_262144 -0.1833 1248478 1019648 BM_StableSort_double_SingleElement_1048576 -0.1741 5703397 4710603 BM_StableSort_double_SingleElement_2097152 -0.1751 10922197 9009835 BM_StableSort_double_SingleElement_4194304 -0.1538 26571923 22485137 BM_StableSort_double_PipeOrgan_1024 -0.4406 10752 6014 BM_StableSort_double_PipeOrgan_4096 -0.5917 49456 20195 BM_StableSort_double_PipeOrgan_16384 -0.6258 270515 101221 BM_StableSort_double_PipeOrgan_65536 -0.7098 1159462 336457 BM_StableSort_double_PipeOrgan_262144 -0.6591 4735711 1614433 BM_StableSort_double_PipeOrgan_1048576 -0.6620 19353110 6541172 BM_StableSort_double_PipeOrgan_2097152 -0.7288 49131812 13323391 BM_StableSort_double_PipeOrgan_4194304 -0.5988 81958974 32878171 BM_StableSort_double_QuickSortAdversary_1024 -0.6516 17948 6254 BM_StableSort_double_QuickSortAdversary_4096 -0.7527 82359 20363 BM_StableSort_double_QuickSortAdversary_16384 -0.7009 340410 101811 BM_StableSort_double_QuickSortAdversary_65536 -0.6854 1487480 467928 BM_StableSort_double_QuickSortAdversary_262144 -0.6386 5648460 2041377 BM_StableSort_double_QuickSortAdversary_1048576 -0.6127 22859142 8852587 BM_StableSort_double_QuickSortAdversary_2097152 -0.7161 68693975 19499381 BM_StableSort_double_QuickSortAdversary_4194304 -0.5909 95532179 39077491 OVERALL_GEOMEAN -0.6472 0 0 ```	2025-04-16 17:34:11 +08:00
A. Jiang	cfa322fa9a	[libc++][NFC] Reuse __bit_log2 for sort (#135303) ... The `__log2i` function template in `<algorithm/sort.h>` is basically equivalent to `__bit_log2` in `<__bit/bit_log2.h>`. It seems better to avoid duplication.	2025-04-13 09:45:18 +08:00
James E T Smith	475cbf0ad6	[libc++] Implement ranges::iota (#68494) ... # Overview As a disclaimer, this is my first PR to LLVM and while I've tried to ensure I've followed the LLVM and libc++ contributing guidelines, there's probably a good chance I missed something. If I have, just let me know and I'll try to correct it as soon as I can. This PR implements `std::ranges::iota` and `std::ranges::out_value_result` outlined in [P2440r1](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2440r1.html). As outlined in the paper above, I've: - Implemented `out_value_result` and added to `<algorithm>` - Added `out_value_result`, `iota_result`, and two overloads of `iota` to `std::ranges` in `<numeric>` - Updated the version macro `__cpp_lib_ranges_iota` in `<version>` I've also added tests for `ranges::iota` and `ranges::out_value_result`. Lastly, I added those structs to the appropriate module files. Partially implements #105184 EDIT: Forgot to mention in the original post, thanks to @hawkinsw for taking a look at a preliminary version of this PR! # TODOs - [x] Updating the range [status doc](https://github.com/jamesETsmith/llvm-project/blob/main/libcxx/docs/Status/RangesMajorFeatures.csv) - [x] Ensure all comments from https://reviews.llvm.org/D121436 are addressed here - [X] EDIT (I'll do this in a separate PR). ~~I'm open to implementing the rest of P2440r1 (`ranges::shift_left` and `ranges::shift_right`) if that's ok, I just wanted to get feedback on `ranges::iota` first~~ - [x] I've been having trouble building the modules locally and want to make sure that's working properly Closes: #134060	2025-04-05 13:46:11 +02:00
Peng Liu	8fdfe3f2a7	[libc++] Refactor ranges::{min, max, min_element, max_element} to use std::__min_element (#132418) ... Previously, ranges::min_element delegated to ranges::__min_element_impl, which duplicated the definition of std::__min_element. This patch updates ranges::min_element to directly call std::__min_element, which allows removing the redundant code in ranges::__min_element_impl. Upon removal of ranges::__min_element_impl, the other ranges algorithms ranges::{min,max,max_element}, which previously delegated to ranges::__min_element_impl, have been updated to call std::__min_element instead. This refactoring unifies the implementation across these algorithms, ensuring that future optimizations or maintenance work only need to be applied in one place.	2025-03-27 09:05:37 -04:00
Louis Dionne	b0668d859b	[libc++] Make sure that __desugars_to isn't tripped up by reference_wrapper, const and ref qualifiers (#132092) ... Previously, const and ref qualification on an operation would cause __desugars_to to report false, which would lead to unnecessary pessimizations. The same holds for reference_wrapper. In practice, const and ref qualifications on the operation itself are not relevant to determining whether an operation desugars to something else or not, so can be ignored. We are not stripping volatile qualifiers from operations in this patch because we feel that this requires additional discussion. Fixes #129312	2025-03-25 14:29:06 -04:00
Louis Dionne	aa80388cf9	[libc++] Ensure that we vectorize algorithms on all Clang-based compilers (#132090) ... Otherwise, we wouldn't vectorize on compilers like AppleClang when in reality we know perfectly well how to do it.	2025-03-24 13:22:40 -04:00
Nikolas Klauser	fb44f00641	[libc++] Add [[gnu::nodebug]] on type traits (#128502)	2025-03-23 21:01:25 +01:00
A. Jiang	053a714add	[libc++] Implement part of P2562R1: constexpr ranges::inplace_merge (#131947) ... Drive-by changes: - Consistently mark `std::__inplace_merge::__inplace_merge_impl` `_LIBCPP_CONSTEXPR_SINCE_CXX26`. - This function template is only called by other functions that becomes constexpr since C++26, and it itself calls `std::__inplace_merge` that is constexpr since C++26. - Unblock related test coverage in constant evaluation for `stable_partition`, `ranges::stable_sort`, `std::stable_sort`, `std::stable_partition`, and `std::inplace_merge`.	2025-03-20 01:59:32 +08:00
Peng Liu	029cb8aff1	[libc++] Fix copy_backward for vector<bool> with small storage types (#131560) ... This patch fixes an issue in libc++ where `std::copy_backward` and `ranges::copy_backward` incorrectly copy `std::vector<bool>` with small storage types (e.g., `uint8_t`, `uint16_t`). The problem arises from flawed bit mask computations involving integral promotions and sign-bit extension, leading to unintended zeroing of bits. This patch corrects the bitwise operations to ensure correct bit-level copying. Fixes #131718.	2025-03-19 11:58:02 -04:00
Peng Liu	a5b3d3a03f	[libc++] Fix {std, ranges}::copy for vector<bool> with small storage types (#131545) ... The current implementation of `{std, ranges}::copy` fails to copy `vector<bool>` correctly when the underlying storage type (`__storage_type`) is smaller than `int`, such as `unsigned char`, `unsigned short`, `uint8_t` and `uint16_t`. The root cause is that the unsigned small storage type undergoes integer promotion to (signed) `int`, which is then left and right shifted, leading to UB (before C++20) and sign-bit extension (since C++20) respectively. As a result, the underlying bit mask evaluations become incorrect, causing erroneous copying behavior. This patch resolves the issue by correcting the internal bitwise operations, ensuring that `{std, ranges}::copy` operates correctly for `vector<bool>` with any custom (unsigned) storage types. Fixes #131692.	2025-03-19 11:55:51 -04:00
Peng Liu	c5195ae2d0	[libc++] Fix {std, ranges}::equal for vector<bool> with small storage types (#130394) ... The current implementation of `{std, ranges}::equal` fails to correctly compare `vector<bool>`s when the underlying storage type is smaller than `int` (e.g., `unsigned char`, `unsigned short`, `uint8_t` and `uint16_t`). See [demo](https://godbolt.org/z/j4s87s6b3)). The problem arises due to integral promotions on the intermediate bitwise operations, leading to incorrect final equality comparison results. This patch fixes the issue by ensuring that `{std, ranges}::equal` operate properly for both aligned and unaligned bits. Fixes #126369.	2025-03-19 11:51:21 -04:00
Peng Liu	e53bea5182	[libc++] Fix ambiguous call in {ranges, std}::count (#122529) ... This PR fixes an ambiguous call encountered while using the `std::ranges::count` and `std::count` algorithms with `vector<bool>` with small `size_type`s. The ambiguity arises from integral promotions during the internal bitwise arithmetic of the `count` algorithms for small integral types. This results in multiple viable candidates: `__libcpp_popcount(unsigned)`,` __libcpp_popcount(unsigned long)`, and `__libcpp_popcount(unsigned long long)`, leading to an ambiguous call error. To resolve this ambiguity, we introduce a dispatcher function, `__popcount`, which directs calls to the appropriate overloads of `__libcpp_popcount`. This closes #122528.	2025-03-19 11:36:29 -04:00
A. Jiang	854a4f2bbb	[libc++] Implement part of P2562R1: constexpr std::inplace_merge (#129008) ... Drive-by: - Adds `constexpr_random.h` for pseudo-randomizing or shuffling in tests for constant evaluation.	2025-03-19 07:42:23 +08:00
Peng Liu	3bd71cbec7	[libc++] Fix ambiguous call in {ranges, std}::find (#122641) ... This PR fixes an ambiguous call encountered when using the `std::ranges::find` or `std::find` algorithms with `vector<bool>` with small `allocator_traits::size_type`s, an issue reported in #122528. The ambiguity arises from integral promotions during the internal bitwise arithmetic of the `find` algorithms when applied to `vector<bool>` with small integral `size_type`s. This leads to multiple viable candidates for small integral types: __libcpp_ctz(unsigned), __libcpp_ctz(unsigned long), and __libcpp_ctz(unsigned long long), none of which represent a single best viable match, resulting in an ambiguous call error. To resolve this, we propose invoking an internal function __countr_zero as a dispatcher that directs the call to the appropriate overload of __libcpp_ctz. Necessary amendments have also been made to __countr_zero.	2025-03-13 14:15:03 -04:00
Peng Liu	4baf1c03fa	[libc++] Optimize ranges::rotate for vector<bool>::iterator (#121168) ... This PR optimizes the performance of `std::ranges::rotate` for `vector<bool>::iterator`. The optimization yields a performance improvement of up to 2096x. Closes #64038.	2025-03-13 14:07:23 -04:00
A. Jiang	ba9aeedf8e	[libc++] Implement part of P2562R1: constexpr ranges::stable_sort (#128860) ... Drive-by: Enables test coverage for `ranges::stable_sort` with proxy iterators, and changes "constexpr in" to "constexpr since" in comments in `<algorithm>`.	2025-03-07 01:27:48 +08:00
A. Jiang	c28c508962	[libc++] Implement part of P2562R1: constexpr ranges::stable_partition (#129839)	2025-03-06 09:23:55 +08:00
Peng Liu	a12744ff05	[libc++] Optimize ranges::swap_ranges for vector<bool>::iterator (#121150) ... This PR optimizes the performance of `std::ranges::swap_ranges` for `vector<bool>::iterator`, addressing a subtask outlined in issue #64038. The optimizations yield performance improvements of up to **611x** for aligned range swap and **78x** for unaligned range swap comparison. Additionally, comprehensive tests covering up to 4 storage words (256 bytes) with odd and even bit sizes are provided, which validate the proposed optimizations in this patch.	2025-03-04 17:15:36 -05:00
A. Jiang	bf9bf291a3	[libc++] Implement part of P2562R1: constexpr std::stable_partition (#128868) ... Drive-by changes: - Enables no-memory case for Clang. - Enables `robust_re_difference_type.compile.pass.cpp` and `robust_against_proxy_iterators_lifetime_bugs.pass.cpp` test coverage for `std::stable_sort` in constant evaluation since C++26. The changes were missing in the PR making `std::stable_sort` `constexpr`.	2025-03-04 09:23:29 +08:00
Nikolas Klauser	1a6f9fd87f	[libc++] Enable algorithm vectorization on arm neon (#128873) ... Previously the wrong detection macro has been used to check whether arm NEON is available. This fixes it, and removes a few unnecessary includes from `__algorithm/simd_utils.h` as a drive-by.	2025-02-28 13:38:52 +01:00
Peng Liu	7717a549e9	[libc++] Optimize ranges::equal for vector<bool>::iterator (#121084) ... This PR optimizes the performance of `std::ranges::equal` for `vector<bool>::iterator`, addressing a subtask outlined in issue #64038. The optimizations yield performance improvements of up to 188x for aligned equality comparison and 82x for unaligned equality comparison. Moreover, comprehensive tests covering up to 4 storage words (256 bytes) with odd and even bit sizes are provided, which validate the proposed optimizations in this patch.	2025-02-26 12:18:25 -05:00
Louis Dionne	5e26fb1699	[libc++] Qualify calls to nullary functions like __throw_foo (#122465) ... This is technically not necessary in most cases to prevent issues with ADL, but let's be consistent. This allows us to remove the libcpp-qualify-declval clang-tidy check, which is now enforced by the robust-against-adl clang-tidy check.	2025-02-21 07:59:46 -05:00
Peng Liu	ab3d793982	[libc++] Optimize ranges::move{,_backward} for vector<bool>::iterator (#121109) ... As a follow-up to #121013 (which optimized `ranges::copy`) and #121026 (which optimized `ranges::copy_backward`), this PR enhances the performance of `std::ranges::{move, move_backward}` for `vector<bool>::iterator`, addressing a subtask outlined in issue #64038. The optimizations bring performance improvements analogous to those achieved for the `{copy, copy_backward}` algorithms: up to 2000x for aligned moves and 60x for unaligned moves. Moreover, comprehensive tests covering up to 4 storage words (256 bytes) with odd and even bit sizes are provided, which validate the proposed optimizations in this patch.	2025-02-19 11:36:45 -05:00
Дмитрий Изволов	6e402f5121	[libc++] Support constexpr for std::stable_sort in radix sort branch (#125284) ... `std::stable_sort` is `constexpr` since PR https://github.com/llvm/llvm-project/pull/110320 But `radix_sort` branch is still non-`constexpr`. This PR fixes it. #119394 #105360	2025-02-06 09:07:25 +01:00
Peng Liu	cf9806eb4d	[libc++] Fix UB in bitwise logic of {std, ranges}::{fill, fill_n} algorithms (#122410) ... This PR addresses an undefined behavior that arises when using the `std::fill` and `std::fill_n` algorithms, as well as their ranges counterparts `ranges::fill` and `ranges::fill_n`, with `vector<bool, Alloc>` that utilizes a custom-sized allocator with small integral types.	2025-02-05 11:39:49 -05:00