KCFI traps should always be recoverable, but as Intrinsic::trap
is marked noreturn, it's not possible to continue execution after
handling the trap as the compiler is free to assume we never
return. Switch to debugtrap instead to ensure we have the option
to resume execution after the trap.
This patch makes the variables names for callback functions more consistent. Changes no functionality.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D151605
This patch implements `__hwasan_memset_match_all`, `__hwasan_memcpy_match_all` and `__hwasan_memmove_match_all`, making hwasan-match-all-tag flag working for hwasan versions of memset, memcpy and memmove.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D149943
Currently, hwasan-match-all-tag flag is supported in inline memory access instrumentation and outline memory access instrumentation, but not supported in callback memory access instrumentation.
- For inline memory access instrumentation: a hwasan-match-all-tag check is added following the tag-mismtach check, if tag from pointer is mismatched with tag from shadow memory and tag from pointer is not equal with hwasan-match-all-tag, then a tag-mismatch will be report.
- For outline memory acess instrumentation: MatchAllTag is encoded in AccessInfo, when emit HWASAN memaccess symbols, asm-printer emits assembly instructions to check if tag from pointer is equal with hwasan-match-all-tag.
- For callback memory access instrumentation: hwasan-match-all-tag check is not implemented in `__hwasan_load`/`__hwasan_store`.
This patch implements a set of callback functions: `__hwasan_[load|store][1|2|4|8|16|n]_match_all` and `__hwasan_load[load|store][1|2|4|8|16|n]_match_all_noabort`, making hwasan-match-all-tag flag working for callback memory access instrumentation.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D149580
The patch supports vp.gather/scatter by allowing addresses being pointer vectors.
And then we just need to check each active pointer element of those pointer vectors.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D149245
This patch adds nosantize metadata to memory access instructions inserted by gcov emitProfileNotes(), making sanitizers skip these instructions when gcov and sanitizer are used together.
Reviewed By: nickdesaulniers
Differential Revision: https://reviews.llvm.org/D150460
The method is marked for deprecation. Delete the method and move all of
its consumers to use the DomTreeUpdater version.
Reviewed By: foad
Differential Revision: https://reviews.llvm.org/D149428
This patch adds a new method setNoSanitizeMetadata() for Instruction, and use it in SanitizerMetadata and SanitizerCoverage.
Reviewed By: nickdesaulniers, MaskRay
Differential Revision: https://reviews.llvm.org/D150632
The patch adds new member MaybeStride into InterestingMemoryOperand to represent
the stride value of experimental.vp.strided.load/store.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D149073
Tag selection for global variables is sequential, starting at a
pseduo-ish seed that's based on the hash of the file name.
Previously, it was possible for a global to be assigned a tag in the
range [1,15]. If the global's size was not a multiple of granules (i.e.
`size % 16 != 0`), then the last granule of the global would be assigned
a short granule tag as well.
If the real memory tag of the global (e.g. '04') happened to collide
with the short granule tag (e.g. '04'), then __hwasan_check would
see that the memory tag matched the short granule tag, and dutifully
assume (in this fast check) that everthing is okay.
Unfortunately, if you tried to access the [5,15]th byte, you never get
to the short granule check. This means you miss intra-granule overflows
on the last granule of a global, if said global was assigned a real
memory tag in the range [1,15].
This causes flakiness in certain global tests, if the SHA of the
filename changes between runs.
This problem also exists for heap and stack allocations as well, but
there's a concern that if we exclude the [1,15] range for heap and stack
that it's an unhappy tradeoff. On Android, this would mean that a 1/255
chance of false positive becomes 1/240. On other platforms though (that
have a less-than-8-bit tag space), this may be unacceptable. We can
think about potential fixes for that in other places, but globals are
fine to reduce the space, as really the only thing that matters is
catching sequential overflow. The false-negative scenario is much, much
more common in use-after-free and use-after-scope, which globals don't
have.
Differential Revision: https://reviews.llvm.org/D150742
Annotation metadata supports adding singular annotation strings to annotation block. This patch adds the ability to insert a tuple of strings into the metadata array.
The idea here is that each tuple of strings represents a piece of information that can be all related. It makes it easier to parse through related metadata information given it will be contained in one tuple.
For example in remarks any pass that implements annotation remarks can have different type of remarks and pass additional information for each.
The original behaviour of annotation remarks is preserved here and we can mix tuple annotations and single annotations for the same instruction.
Reviewed By: paquette
Differential Revision: https://reviews.llvm.org/D148328
The patch adds new member MaybeEVL into InterestingMemoryOperand to represent
the effective vector length for vp intrinsics. It may be extended for some target intrinsics in the future.
Reviewed By: kito-cheng
Differential Revision: https://reviews.llvm.org/D146208
Now that the runtime tracks the lifetime access density directly, we can
use that directly in the threshold checks instead of less accurately
computing from other statistics.
Differential Revision: https://reviews.llvm.org/D149684
This commit removes potential UB in the PGO instrumentation passes that
was caused by casting away constness and then potentially modifying the
object.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D148903
This is folloup to b5595836, which missed the
Replacemen variable.
Before b5595836 the code assumed that alloca
ptrs are not tagged so tagging is implemented
as simple OR.
So this patch completes support of tagged SP
by passing untagged alloca pointers into
tagPointer.
c55fffe replaced fcmp with fpclass.
```
declare i1 @llvm.is.fpclass(<fptype> <op>, i32 <test>)
declare <N x i1> @llvm.is.fpclass(<vector-fptype> <op>, i32 <test>)
```
Perfect fix will require checking bits of <op> corresponding to <test>
argument. For now just propagate shadow without reporting before
intrinsic. Still existing handling of fcmp is also simple OR, so it's
not making it worse.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D149491
This commit cleans up some parts of the PGO instrumentation. Most
importantly, it removes a template parameter shadowing of a class name
that could lead to confusion.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D149324
This commit moves the CFGMST.h file into the include directory. The
implemented algorithm is can be helpful for downstream projects that
want to use the PGO data in a non-standard way.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D149336
Enable -fsanitize=kernel-memory support in Clang.
The x86_64 ABI requires that shadow_origin_ptr_t must be returned via a
register pair, and the s390x ABI requires that it must be returned via
memory pointed to by a hidden parameter. Normally Clang takes care of
the ABI, but the sanitizers run long after it, so unfortunately they
have to duplicate the ABI logic.
Therefore add a special case for SystemZ and manually emit the
s390x-ABI-compliant calling sequences. Since it's only 2 architectures,
do not create a VarArgHelper-like abstraction layer.
The kernel functions are compiled with the "packed-stack" and
"use-soft-float" attributes. For the "packed-stack" functions, it's not
correct for copyRegSaveArea() to copy 160 bytes of shadow and origins,
since the save area is dynamically sized. Things are greatly simplified
by the fact that the vararg "use-soft-float" functions use precisely
56 bytes in order to save the argument registers to where va_arg() can
find them.
Make copyRegSaveArea() copy only 56 bytes in the "use-soft-float" case.
The "packed-stack" && !"use-soft-float" case has no practical uses at
the moment, so leave it for the future.
Add tests.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D148596
If stack was allocated using regular allocator, it may be tagged
and it will make memToShadow calculate invalid offset.
Also when UAR tag should be the tag of the stack frame pointer.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D149228
This is leftover from older version of HWASAN.
The current HWASAN assumes that the new stack
frames are tagged with zeroes, which make getNextTagWithCall
or StackTag ^ TagMaskByte unusable.
Reviewed By: kstoimenov, eugenis
Differential Revision: https://reviews.llvm.org/D149220
Remove C APIs for interacting with PassRegistry and pass
initialization. These are legacy PM concepts, and are no longer
relevant for the new pass manager.
Calls to these initialization functions can simply be dropped.
Differential Revision: https://reviews.llvm.org/D145043
As described in [0], this extends IRPGO to support //Temporal Profiling//.
When `-pgo-temporal-instrumentation` is used we add the `llvm.instrprof.timestamp()` intrinsic to the entry of functions which in turn gets lowered to a call to the compiler-rt function `INSTR_PROF_PROFILE_SET_TIMESTAMP()`. A new field in the `llvm_prf_cnts` section stores each function's timestamp. Then in `llvm-profdata merge` we convert these function timestamps into a //trace// and add it to the indexed profile.
Since these traces could significantly increase the profile size, we've added `-max-temporal-profile-trace-length` and `-temporal-profile-trace-reservoir-size` to limit the length of a trace and the number of traces in a profile, respectively.
In a future diff we plan to use these traces to construct an optimized function order to reduce the number of page faults during startup.
Special thanks to Julian Mestre for helping with reservoir sampling.
[0] https://discourse.llvm.org/t/rfc-temporal-profiling-extension-for-irpgo/68068
Reviewed By: snehasish
Differential Revision: https://reviews.llvm.org/D147287
Ironically, MSan copies uninitialized data off the stack into
VAArgTLSCopy in the callee-side handling of va_start. Clamp the copy
size to the actual length of the buffer, and zero-initialize the
remainder.
Differential Revision: https://reviews.llvm.org/D146858
This diff implements minimal block coverage instrumentation. When the `-pgo-block-coverage` option is used, basic blocks will be instrumented for block coverage using single byte booleans. The coverage of some basic blocks can be inferred from others, so not every basic block is instrumented. In fact, we found that only ~60% of basic blocks need to be instrumented. These differences lead to less size overhead when compared to instrumenting block counts. For example, block coverage on the clang binary has an overhead of 20 Mi (17%) compared to 56 Mi (47%) with block counts.
Even though block coverage profiles have less precision than block count profiles, they can still be used to guide optimizations. In `PGOUseFunc` we use block coverage to populate edge weights such that BFI gives nonzero counts to only covered blocks. We do this by 1) setting the entry count of covered functions to a large value, i.e., 10000 and 2) populating edge weights using block coverage. In the next diff https://reviews.llvm.org/D125743 we use BFI to guide the machine outliner to avoid outlining covered blocks. This `-pgo-block-coverage` option provides a trade off of generating less precise profiles for faster and smaller instrumented binaries.
The `BlockCoverageInference` class defines the algorithm to find the minimal set of basic blocks that need to be instrumented for coverage. This is different from the Kirchhoff circuit law optimization that is used for edge **counts** because that does not work for block **coverage**. The reason for this is that edge counts can be added together to find a missing count while block coverage cannot since they store boolean values. So we need a new algorithm to find which blocks must be instrumented.
The details on this algorithm can be found in this paper titled "Minimum Coverage Instrumentation": https://arxiv.org/abs/2208.13907
Special thanks to Julian Mestre for creating this block coverage inference algorithm.
Binary size of `clang` using `-O2`:
* Base
* `.text`: 65.8 Mi
* Total: 119 Mi
* IRPGO (`-fprofile-generate -mllvm -disable-vp -mllvm -debug-info-correlate`)
* `.text`: 93.0 Mi
* `__llvm_prf_cnts`: 14.5 Mi
* Total: 175 Mi
* Minimal Block Coverage (`-fprofile-generate -mllvm -disable-vp -mllvm -debug-info-correlate -mllvm -pgo-block-coverage`)
* `.text`: 82.1 Mi
* `__llvm_prf_cnts`: 1.38 Mi
* Total: 139 Mi
Reviewed By: spupyrev, kyulee
Differential Revision: https://reviews.llvm.org/D124490
Many uses of getIntPtrType() were using that type to calculate the
neened type for GEP offset arguments. However, some time ago,
DataLayout was extended to support pointers where the size of the
pointer is not equal to the size of the values used to index it.
Much code was already migrated to, for example, use getIndexSizeInBits
instead of getPtrSizeInBits, but some rewrites still used
getIntPtrType() to get the type for GEP offsets.
This commit changes uses of getIntPtrType() to getIndexType() where
they are involved in a GEP-related calculation.
In at least one case (bounds check insertion) this resolves a compiler
crash that the new test added here would previously trigger.
This commit does not impact
- C library-related rewriting (memcpy()), which are operating under
the assumption that intptr_t == size_t. While all the mechanisms for
breaking this assumption now exist, doing so is outside the scope of
this commit.
- Code generation and below. Note that the use of getIntPtrType() in
CodeGenPrepare will be changed in a future commit.
- Usage of getIntPtrType() in any backend
Depends on D143435
Reviewed By: arichardson
Differential Revision: https://reviews.llvm.org/D143437
This adds support for scalable vector types - at least far enough to get basic load and store cases working. It turns out that load/store without origin tracking already worked; I apparently got that working with one of the pre patches to use TypeSize utilities and didn't notice. The code changes here are required to enable origin tracking.
For origin tracking, a 4 byte value - the origin - is broadcast into a shadow region whose size exactly matches the type being accessed. This origin is only written if the shadow value is non-zero. The details of how shadow is computed from the original value being stored aren't relevant for this patch.
The code changes involve two related primitives.
First, we need to be able to perform that broadcast into a scalable sized memory region. This requires the use of a loop, and appropriate bound. The fixed size case optimizes with larger stores and alignment; I did not bother with that for the scalable case for now. We can optimize this codepath later if desired.
Second, we need a way to test if the shadow is zero. The mechanism for this in the code is to convert the shadow value into a scalar, and then zero check that. There's an assumption that this scalar is zero exactly when all elements of the shadow value are zero. As a result, we use an OR reduction on the scalable vector. This is analogous to how e.g. an array is handled. I landed a bunch of cleanup changes to remove other direct uses of the scalar conversion to convince myself there were no other undocumented invariants.
Differential Revision: https://reviews.llvm.org/D146157
