This releases the requirement that we need to preserve the memory for
all regions at the beginning. It needs a huge amount of contiguous pages
and which may be a challenge in certain cases. Therefore, adding a new
flag, EnableContiguousRegions, to indicate whether we want to allocate
all the regions next to each other.
Note that once the EnableContiguousRegions is disabled,
EnableRandomOffset becomes irrelevant because the base of each region is
already random.
Instead of explicitly disabling a feature by declaring the variable and
set it to false, this change supports the optional flags. I.e., you can
skip certain flags if you are not using it.
This optional feature supports both forms,
1. Value: A parameter for a feature. E.g., EnableRandomOffset
2. Type: A C++ type implementing a feature. E.g., ConditionVariableT
On the other hand, to access the flags will be through one of the
wrappers, BaseConfig/PrimaryConfig/SecondaryConfig/CacheConfig
(CacheConfig is embedded in SecondaryConfig). These wrappers have the
getters to access the value and the type. When adding a new feature, we
need to add it to `allocator_config.def` and mark the new variable with
either *_REQUIRED_* or *_OPTIONAL_* macro so that the accessor will be
generated properly.
In addition, also remove the need of `UseConditionVariable` to flip
on/off of condition variable. Now we only need to define the type of
condition variable.
Reverts llvm/llvm-project#70390
There's a bug caught by
`ScudoCombinedTestReallocateInPlaceStress_DefaultConfig.ReallocateInPlaceStress`
with gwp asan. It's an easy fix but given that this is a major change, I
would like to revert it first
Instead of always storing the same number of blocks as cached, we prefer
increasing the utilization by saving more blocks in a single
TransferBatch. This may slightly impact the performance, but it will
save a lot of memory used by BatchClassId (especially for larger
blocks).
This may improve the waiting of `Region->MMLock` while trying to refill
the freelist. Instead of always waiting on the completion of
`populateFreeListAndPopBatch()` or `releaseToOSMaybe()`, `pushBlocks()`
also refills the freelist. This increases the chance of earlier return
from `popBatches()`.
The support of condition variable hasn't been done for all platforms.
Therefore, add another `popBatchWithCV()` and it can be configured in
the allocator configuration by setting `Primary::UseConditionVariable`
and the desired `ConditionVariableT`.
Reviewed By: cferris
Differential Revision: https://reviews.llvm.org/D156146
This change moves the `TransferBatch` and `BatchGroup` out of
SizeClassAllocatorLocalCache. It allows us that the node in freelist can
store more blocks instead of depending on the number of blocks cached.
That means we will be able to store more blocks in each node of freelist
and therefore reduce the memory used by BatchClass (with little
performance overhead). Note that we haven't enabled that in this patch.
This is the first step of this transition.
This config is not actually used anywhere and it is not used on Android.
Since it does not test anything not tested elsewhere, remove it.
Remove the size class data associated with this config too.
Modify the tests so that all clang warnings can be turned up to high.
Fix all places flagged by -Wconversion.
Fix a few unused variables not marked with UNUSED.
For the memtag testing, only compile some tests for 64 bit since
compiling them on 32 bit leads to warnings/errors. All of the tests
are already skipped on 32 bit OSes, so this will not affect any
real tests.
Reviewed By: Chia-hungDuan
Differential Revision: https://reviews.llvm.org/D155749
When all the blocks (local caches are included) are freed, the size of
free blocks should be equal to `AllocatedUser`.
Reviewed By: cferris
Differential Revision: https://reviews.llvm.org/D152769
To define custom allocation, you only need to put the configuration in
custom_scudo_config.h and define two required aliases, then you will be
switched to the customized config and the tests will also run with your
configuration.
In this CL, we also have a minor refactor the structure of
configuration. Now the essential fields are put under the associated
hierarchy and which will make the defining new configuration easier.
Reviewed By: cferris
Differential Revision: https://reviews.llvm.org/D150481
Release pages for large block (size greater than a page) is faster than
the small blocks. Besides, larger blocks are supposed not to be used
so often like smaller blocks which means we may hold several pages used
by large block and rarely get chance to release them if there's no
explicit M_PURGE call. Therefore, relax the release-interval condition
for large block.
This also fixes the assumption that FORCE_ALL should always try page
release.
Differential Revision: https://reviews.llvm.org/D151290
The force flag to releaseToOSMaybe does not release everything
since it is an expensive operation. Modify the release flag to
have three states: normal, force, forceall. Force behaves the same
as setting Force to true from before this change. Forceall will
release everything regardless of how much time it takes, or
how much there is to release.
In addition, add a new mallopt that will call the release function
with the forceall flag set.
Reviewed By: Chia-hungDuan
Differential Revision: https://reviews.llvm.org/D146106
This is not a pure revert of c929bcb7d85700494217f3a2148549f8757e0eed.
It also includes a bug fix.
Differential Revision: https://reviews.llvm.org/D136029
In SizeClassAllocator64, the RegionBeg is determined by RegionBase +
random offset. The offset is n pages, where n is a random number less or
equal to 16. However, on certain platforms which have large page size,
it may end up immediately OOM without mapping any block pages. For
example,
PageSize = 64 KB, RegionSize = 1 MB
Suppose the random number n is 16, then the random offset will be
64 * 16 = 1024 KB which is equal to the RegionSize.
On most platforms we don't have such large page size and we have
different PRNG(pseudo random number generator) behaviors, thus we didn't
hit any failures before. Given that this now only affects the tests,
only increase the region size is enough.
Will revisit the logic of calculating the random offset.
Differential Revision: https://reviews.llvm.org/D136025
Scudo is supposed to allocate any blocks across the entired mapped
pages and each page is equally likely to be selected. Which means Scudo
is leaning to touch as many pages as possible. This brings better
security but it also sacrifices the chance of releasing dirty pages.
To alleviate the unmanagable footprint growing, this CL introduces the
BatchGroup concept. Each blocks will be classified into a BatchGroup
according to its address. While allocation, we are leaning to allocate
blocks in the same group first. Note that the blocks selected from a
group is still random over several pages. At the same time, we have
better prediction of dirty page growing speed. Besides, we are able to
do partial page releasing by examing part of BatchGroups.
Reviewed By: cryptoad, cferris
Differential Revision: https://reviews.llvm.org/D133897
The Count/MaxCount used in TransferBatch and PerClass can be fit in u16 in
current configurations and it's also reasonable to have a u16 limit. The
spare 16 bits will be used for additional status like pages mapping
status in a TransferBatch.
Reviewed By: cryptoad, cferris, vitalybuka
Differential Revision: https://reviews.llvm.org/D133145
Explicitly specify the class name to avoid selecting the wrong Run function, and inherit from the correct Test parent
Differential Revision: https://reviews.llvm.org/D121854
trusty.cpp and trusty.h define Trusty implementations of map and other
platform-specific functions. In addition to adding Trusty configurations
in allocator_config.h and size_class_map.h, MapSizeIncrement and
PrimaryEnableRandomOffset are added as configurable options in
allocator_config.h.
Background on Trusty: https://source.android.com/security/trusty
Differential Revision: https://reviews.llvm.org/D103578
Some platforms (eg: Trusty) are extremelly memory constrained, which
doesn't necessarily work well with some of Scudo's current assumptions.
`Vector` by default (and as such `String` and `ScopedString`) maps a
page, which is a bit of a waste. This CL changes `Vector` to use a
buffer local to the class first, then potentially map more memory if
needed (`ScopedString` currently are all stack based so it would be
stack data). We also want to allow a platform to prevent any dynamic
resizing, so I added a `CanGrow` templated parameter that for now is
always `true` but would be set to `false` on Trusty.
Differential Revision: https://reviews.llvm.org/D103641
Operator new must align allocations for types with large alignment.
Before c++17 behavior was implementation defined and both clang and gc++
before 11 ignored alignment. Miss-aligned objects mysteriously crashed
tests on Ubuntu 14.
Alternatives are compile with -std=c++17 or -faligned-new, but they were
discarded as less portable.
Reviewed By: hctim
Differential Revision: https://reviews.llvm.org/D101874
This CL introduces configuration options to allow pointers to be
compacted in the thread-specific caches and transfer batches. This
offers the possibility to have them use 32-bit of space instead of
64-bit for the 64-bit Primary, thus cutting the size of the caches
and batches by nearly half (and as such the memory used in size
class 0). The cost is an additional read from the region information
in the fast path.
This is not a new idea, as it's being used in the sanitizer_common
64-bit primary. The difference here is that it is configurable via
the allocator config, with the possibility of not compacting at all.
This CL enables compacting pointers in the Android and Fuchsia default
configurations.
Differential Revision: https://reviews.llvm.org/D96435
Said test was flaking on Fuchsia for non-obvious reasons, and only
for ASan variants (the release was returning 0).
It turned out that the templating was off, `true` being promoted to
a `s32` and used as the minimum interval argument. This meant that in
some circumstances, the normal release would occur, and the forced
release would have nothing to release, hence the 0 byte released.
The symbols are giving it away (note the 1):
```
scudo::SizeClassAllocator64<scudo::FixedSizeClassMap<scudo::DefaultSizeClassConfig>,24ul,1,2147483647,false>::releaseToOS(void)
```
This also probably means that there was no MTE version of that test!
Differential Revision: https://reviews.llvm.org/D88457
Summary:
Partners have requested the ability to configure more parts of Scudo
at runtime, notably the Secondary cache options (maximum number of
blocks cached, maximum size) as well as the TSD registry options
(the maximum number of TSDs in use).
This CL adds a few more Scudo specific `mallopt` parameters that are
passed down to the various subcomponents of the Combined allocator.
- `M_CACHE_COUNT_MAX`: sets the maximum number of Secondary cached items
- `M_CACHE_SIZE_MAX`: sets the maximum size of a cacheable item in the Secondary
- `M_TSDS_COUNT_MAX`: sets the maximum number of TSDs that can be used (Shared Registry only)
Regarding the TSDs maximum count, this is a one way option, only
allowing to increase the count.
In order to allow for this, I rearranged the code to have some `setOption`
member function to the relevant classes, using the `scudo::Option` class
enum to determine what is to be set.
This also fixes an issue where a static variable (`Ready`) was used in
templated functions without being set back to `false` every time.
Reviewers: pcc, eugenis, hctim, cferris
Subscribers: jfb, llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D84667
When the hardware and operating system support the ARM Memory Tagging
Extension, tag primary allocation granules with a random tag. The granules
either side of the allocation are tagged with tag 0, which is normally
excluded from the set of tags that may be selected randomly. Memory is
also retagged with a random tag when it is freed, and we opportunistically
reuse the new tag when the block is reused to reduce overhead. This causes
linear buffer overflows to be caught deterministically and non-linear buffer
overflows and use-after-free to be caught probabilistically.
This feature is currently only enabled for the Android allocator
and depends on an experimental Linux kernel branch available here:
https://github.com/pcc/linux/tree/android-experimental-mte
All code that depends on the kernel branch is hidden behind a macro,
ANDROID_EXPERIMENTAL_MTE. This is the same macro that is used by the Android
platform and may only be defined in non-production configurations. When the
userspace interface is finalized the code will be updated to use the stable
interface and all #ifdef ANDROID_EXPERIMENTAL_MTE will be removed.
Differential Revision: https://reviews.llvm.org/D70762
Summary:
This CL makes unit tests compatible with Fuchsia's zxtest. This
required a few changes here and there, but also unearthed some
incompatibilities that had to be addressed.
A header is introduced to allow to account for the zxtest/gtest
differences, some `#if SCUDO_FUCHSIA` are used to disable incompatible
code (the 32-bit primary, or the exclusive TSD).
It also brought to my attention that I was using
`__scudo_default_options` in different tests, which ended up in a
single binary, and I am not sure how that ever worked. So move
this to the main cpp.
Additionally fully disable the secondary freelist on Fuchsia as we do
not track VMOs for secondary allocations, so no release possible.
With some modifications to Scudo's BUILD.gn in Fuchsia:
```
[==========] 79 tests from 23 test cases ran (10280 ms total).
[ PASSED ] 79 tests
```
Reviewers: mcgrathr, phosek, hctim, pcc, eugenis, cferris
Subscribers: srhines, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70682
Summary:
Following up on D68471, this CL introduces some `getStats` APIs to
gather statistics in char buffers (`ScopedString` really) instead of
printing them out right away. Ultimately `printStats` will just
output the buffer, but that allows us to potentially do some work
on the intermediate buffer, and can be used for a `mallocz` type
of functionality. This allows us to pretty much get rid of all the
`Printf` calls around, but I am keeping the function in for
debugging purposes.
This changes the existing tests to use the new APIs when required.
I will add new tests as suggested in D68471 in another CL.
Reviewers: morehouse, hctim, vitalybuka, eugenis, cferris
Reviewed By: morehouse
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D68653
llvm-svn: 374173
Summary:
There was an issue in `releaseToOSMaybe`: one of the criteria to
decide if we should proceed with the release was wrong. Namely:
```
const uptr N = Sci->Stats.PoppedBlocks - Sci->Stats.PushedBlocks;
if (N * BlockSize < PageSize)
return; // No chance to release anything.
```
I meant to check if the amount of bytes in the free list was lower
than a page, but this actually checks if the amount of **in use** bytes
was lower than a page.
The correct code is:
```
const uptr BytesInFreeList =
Region->AllocatedUser -
(Region->Stats.PoppedBlocks - Region->Stats.PushedBlocks) * BlockSize;
if (BytesInFreeList < PageSize)
return 0; // No chance to release anything.
```
Consequences of the bug:
- if a class size has less than a page worth of in-use bytes (allocated
or in a cache), reclaiming would not occur, whatever the amount of
blocks in the free list; in real world scenarios this is unlikely to
happen and be impactful;
- if a class size had less than a page worth of free bytes (and enough
in-use bytes, etc), then reclaiming would be attempted, with likely
no result. This means the reclaiming was overzealous at times.
I didn't have a good way to test for this, so I changed the prototype
of the function to return the number of bytes released, allowing to
get the information needed. The test added fails with the initial
criteria.
Another issue is that `ReleaseToOsInterval` can actually be 0, meaning
we always try to release (side note: it's terrible for performances).
so change a `> 0` check to `>= 0`.
Additionally, decrease the `CanRelease` threshold to `PageSize / 32`.
I still have to make that configurable but I will do it at another time.
Finally, rename some variables in `printStats`: I feel like "available"
was too ambiguous, so change it to "total".
Reviewers: morehouse, hctim, eugenis, vitalybuka, cferris
Reviewed By: morehouse
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D68471
llvm-svn: 373930
