This patch adds a new `TargetLowering` hook `lowerEHPadEntry()` that is
called at the start of lowering EH pads in SelectionDAG. This allows the
insertion of target-specific actions on entry to exception handlers.
This is used on AArch64 to insert SME streaming-mode switches at landing
pads. This is needed as exception handlers are always entered with
PSTATE.SM off, and the function needs to resume the streaming mode of
the function body.
Support tail calls to whole wave functions (trivial) and from whole wave
functions (slightly more involved because we need a new pseudo for the
tail call return, that patches up the EXEC mask).
Move the expansion of whole wave function return pseudos (regular and
tail call returns) to prolog epilog insertion, since that's where we
patch up the EXEC mask.
This hook replaces inline asm with LLVM intrinsics. It was intended to
match inline assembly implementations of bswap in libc headers and
replace them more optimizable implementations.
At this point, it has outlived its usefulness (see
https://github.com/llvm/llvm-project/issues/156571#issuecomment-3247638412),
as libc implementations no longer use inline assembly for this purpose.
Additionally, it breaks the "black box" property of inline assembly,
which some languages like Rust would like to guarantee.
Fixes https://github.com/llvm/llvm-project/issues/156571.
This patch implements direct N-way splitting for wide scalar shifts
instead
of recursive binary splitting. For example, an i512 G_SHL can now be
split
directly into 8 i64 operations rather than going through i256 -> i128 ->
i64.
The main motivation behind this is to alleviate (although not entirely
fix)
pathological compile time issues with huge types, like i4224. The
problem
we see is that the recursive splitting strategy combined with our messy
artifact combiner ends up with terribly long compiles as tons of
intermediate
artifacts are generated, and then attempted to be combined ad-nauseum.
Going directly from the large shifts to the destination types
short-circuits
a lot of these issues, but it's still an abuse of the backend and
front-ends
should never be doing this sort of thing.
RISCV does not provide a native atomic subtract instruction, so this
patch lowers `G_ATOMICRMW_SUB` by negating the RHS value and performing
an atomic add. The legalization rules in `RISCVLegalizerInfo` are
updated accordingly, with libcall fallbacks when `StdExtA` is not
available, and intrinsic legalization is extended to support
`riscv_masked_atomicrmw_sub`.
For example, lowering
`%1 = atomicrmw sub ptr %a, i32 1 seq_cst`
on riscv32a produces:
```
li a1, -1
amoadd.w.aqrl a0, a1, (a0)
```
On riscv64a, where the RHS type is narrower than XLEN, it currently
produces:
```
li a1, 1
neg a1, a1
amoadd.w.aqrl a0, a1, (a0)
```
There is still a constant-folding or InstConbiner gap. For instance,
lowering
```
%b = sub i32 %x, %y
%1 = atomicrmw sub ptr %a, i32 %b seq_cst
```
generates:
```
subw a1, a1, a2
neg a1, a1
amoadd.w.aqrl a0, a1, (a0)
```
This sequence could be optimized further to eliminate the redundant neg.
Addressing this may require improvements in the Combiner or Peephole
Optimizer in future work.
---------
Co-authored-by: Kane Wang <kanewang95@foxmail.com>
This patch implements a correctly rounded expansion of the frem
instruction in LLVM IR. This is useful for target architectures for
which such an expansion is too involved to be implement in ISel
Lowering. The expansion is based on the code from the AMD device libs
and has been tested successfully against the OpenCL conformance tests on
amdgpu. The expansion is implemented in the preexisting "expand-fp"
pass. It replaces the expansion of "frem" in ISel for the amdgpu target;
it is enabled for targets which do not directly support "frem" and for
which no matching "fmod" LibCall is available.
---------
Co-authored-by: Matt Arsenault <Matthew.Arsenault@amd.com>
In the InterleavedAccessPass the function getMask assumes that
shufflevector operations are always fixed width, which isn't true
because we use them for splats of scalable vectors. This patch fixes the
code by bailing out for scalable vectors.
This patch changes the findSingleRegDef function from si-peephole-sdwa
to reuse MachineRegisterInfo::getOneDef and findSingleRefUse to use a
new MachineRegisterInfo::getOneNonDBGUse function.
As noted in #153256, TableGen is generating reserved names for
RuntimeLibcalls, which resulted in a build failure for Arm64EC since
`vcruntime.h` defines `__security_check_cookie` as a macro.
To avoid using reserved names, all impl names will now be prefixed with
`Impl_`.
`NumLibcallImpls` was lifted out as a `constexpr size_t` instead of
being an enum field.
While I was churning the dependent code, I also removed the TODO to move
the impl enum into its own namespace and use an `enum class`: I
experimented with using an `enum class` and adding a namespace, but we
decided it was too verbose so it was dropped.
It can be unsafe to load a vector from an address and write a vector to
an address if those two addresses have overlapping lanes within a
vectorised loop iteration.
This PR adds intrinsics designed to create a mask with lanes disabled if
they overlap between the two pointer arguments, so that only safe lanes
are loaded, operated on and stored. The `loop.dependence.war.mask`
intrinsic represents cases where the store occurs after the load, and
the opposite for `loop.dependence.raw.mask`. The distinction between
write-after-read and read-after-write is important, since the ordering
of the read and write operations affects if the chain of those
instructions can be done safely.
Along with the two pointer parameters, the intrinsics also take an
immediate that represents the size in bytes of the vector element types.
This will be used by #100579.
The patch add branch hint for AtomicExpandImpl::expandAtomicCmpXchg, For
example: in PowerPC, it support branch hint as
```
loop:
lwarx r6,0,r3 # load and reserve
cmpw r4,r6 #1st 2 operands equal? bne- exit #skip if not
bne- exit #skip if not
stwcx. r5,0,r3 #store new value if still res’ved bne- loop #loop if lost reservation
bne- loop #loop if lost reservation
exit:
mr r4,r6 #return value from storage
```
`-` hints not taken,
`+` hints taken,
This patch is motivated by
https://github.com/llvm/llvm-project/pull/149827, where we plan on using
mangled names on structor declarations to find the exact structor
definition that LLDB's expression evaluator should call.
So far LLVM expects the declaration and definition linkage names to be
identical (or the declaration to just not have a linkage name). But we
plan on attaching the GCC-style "unified" mangled name to declarations,
which will be different to linkage name on the definition. This patch
relaxes this restriction.
Instead of std::optional<uint64_t>. Shift amounts must be less than or
equal to our maximum supported bit widths which fit in unsigned. Most of
the callers already assumed it fit in unsigned.
Backends like NVPTX use -1 to indicate `true` and 0 to indicate `false`
for boolean values. Machine instruction `#DBG_VALUE` also uses -1 to
indicate a `true` boolean constant.
However, during the DWARF generation, booleans are treated as unsigned
variables, and the debug_loc expression, like `DW_OP_lit0; DW_OP_not` is
emitted for the `true` value.
This leads to the debugger printing `255` instead of `true` for constant
boolean variables.
This change emits `DW_OP_lit1` instead of `DW_OP_lot0; DW_OP_not`.
The test llvm/test/DebugInfo/X86/pr12831.ll was added in 4d358b55fa to
fix the issue with emission of empty DW_TAG_subprogram tags
(https://bugs.llvm.org/show_bug.cgi?id=12831).
However, the test output is not checked properly, and it contains:
```
0x00000206: DW_TAG_subprogram
0x00000207: DW_TAG_reference_type
DW_AT_type (0x00000169 "class ")
```
The reason is that the DIE for the definition DISubprogram "writeExpr"
is created during the call to `getOrCreateSubprogramDIE(declaration of
writeExpr)`. Therefore, when `getOrCreateSubprogramDIE(definition of
writeExpr)` is first called, we get a recursive chain of calls:
```
getOrCreateSubprogramDIE(definition of writeExpr)
getOrCreateSubprogramDIE(declaration of writeExpr)
...
getOrCreateSubprogramDIE(definition of writeExpr)
```
The outer call doesn't expect that the DIE for the definition of
writeExpr will be created during the creation of declaration DIE. So,
another DIE is created for the same subprogram. In this PR, a check is
added to fix that.
closes#154331
This PR addresses all minimum changes needed to compile LLVM and MLIR
with the c++23 standard.
It is a work in progress and to be reviewed for better methods of
handling the parts of the build broken by c++23.
Remove "approx-func-fp-math" attribute and related command line option,
users should always use afn flag in IR.
Resolve FIXME in `TargetMachine::resetTargetOptions` partially.
Loads didn't have the `Expand` option in `AtomicExpandPass`. Stores had
`Expand` but it didn't defer to TLI and instead did an action directly.
Add a `CustomExpand` option and make it always map to the TLI hook for
all cases. The `Expand` option now refers to a generic expansion for all
targets.
After #139914, `DIBilder::finalize()` finalizes both declaration and
definition DISubprograms.
Therefore, there is no need to call `DIBuilder::finalizeSubprogram()`
right before `DIBilder::finalize()`.
This Adds scalarization handling for fewer vector elements of insert and
extract, so that i128 and fp128 types can be handled if they make it
past combines. Inserts are unmerged with the inserted element added to
the remerged vector, extracts are unmerged then the correct element is
copied into the destination. With a non-constant vector the usual stack
lowering is used.
When tracking defs in finalizeBundle two sets are used. LocalDefs is
used to track defined virtual and physical registers, while LocalDefsP
is used to track defined register units for the physical registers.
This patch moves the updates of LocalDefsP to only iterate over regunits
when a new physical register is added to LocalDefs. When the physical
register already is present in LocalDefs, then the corresponding
register units are present in LocalDefsP. So it was a waste of time to
add them to the set again.
Closes https://github.com/llvm/llvm-project/issues/155345.
In the original case, we have one frozen use and two unfrozen uses:
```
t73: i8 = select t81, Constant:i8<0>, t18
t75: i8 = select t10, t18, t73
t59: i8 = freeze t18 (combining)
t80: i8 = freeze t59 (another user of t59)
```
In `DAGCombiner::visitFREEZE`, we replace all uses of `t18` with `t59`.
After updating the uses, `t59: i8 = freeze t18` will be updated to `t59:
i8 = freeze t59` (`AddModifiedNodeToCSEMaps`) and CSEed into `t80: i8 =
freeze t59` (`ReplaceAllUsesWith`). As the previous call to
`AddModifiedNodeToCSEMaps` already removed `t59` from the CSE map,
`ReplaceAllUsesWith` cannot remove `t59` again.
For clarity, see the following call graph:
```
ReplaceAllUsesOfValueWith(t18, t59)
ReplaceAllUsesWith(t18, t59)
RemoveNodeFromCSEMaps(t73)
update t73
AddModifiedNodeToCSEMaps(t73)
RemoveNodeFromCSEMaps(t75)
update t75
AddModifiedNodeToCSEMaps(t75)
RemoveNodeFromCSEMaps(t59) <- first delection
update t59
AddModifiedNodeToCSEMaps(t59)
ReplaceAllUsesWith(t59, t80)
RemoveNodeFromCSEMaps(t59) <- second delection
Boom!
```
This patch unfreezes all the uses first to avoid triggering CSE when
introducing cycles.
When going through the ISD::EXTRACT_ELEMENT case, `KnownSign - rIndex *
BitWidth`
could produce a negative. When a negative is produced, the lower bound
of the `std::clamp` is returned. Change that lower bound to one to avoid
potential underflows, because the expectation is that
`ComputeNumSignBits`
should always return at least 1.
Fixes#155452.
This is a sibling patch to #151612: passing gap masks to the renewal TLI
hooks for lowering interleaved stores that use shufflevector to do the
interleaving.
If the srl+shl have the same shift amount and the shl has the nuw flag,
we can remove both.
In the affected test, the InterleavedAccess pass will emit a udiv after
the `mul nuw`. We expect them to combine away. The remaining shifts on
the RV64 tests are because we didn't add the zeroext attribute to the
incoming evl operand.
The `getDefSrcRegIgnoringCopies` method in GlobalISel Utils crashed when
the first operand of the input instruction was not a register, e.g.,
the `INLINEASM` instruction has a non-register first operand.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
This PR simply moves the callsite anchors from the beginning of
callsites to their end.
Emitting the end of callsites is more sensible as it allows breaking the
basic block into subblocks which end with control transfer instructions.
The original version of this change inadvertently dropped
b6e19b35cd87f3167a0f04a61a12016b935ab1ea. This version retains that fix
as well as adding tests for it and an explanation for why it is needed.
This is done for consistency with LiveRegUnits (see #154325). This is
technically not an NFC, as `MBB.liveouts()` excludes runtime-defined
liveins, but no users currently depend on this.
Add entries for_stack_chk_guard, __ssp_canary_word, __security_cookie,
and __guard_local. As far as I can tell these are all just different
names for the same shaped functionality on different systems.
These aren't really functions, but special global variable names. They
should probably be treated the same way; all the same contexts that
need to know about emittable function names also need to know about
this. This avoids a special case check in IRSymtab.
This isn't a complete change, there's a lot more cleanup which
should be done. The stack protector configuration system is a
complete mess. There are multiple overlapping controls, used in
3 different places. Some of the target control implementations overlap
with conditions used in the emission points, and some use correlated
but not identical conditions in different contexts.
i.e. useLoadStackGuardNode, getIRStackGuard, getSSPStackGuardCheck and
insertSSPDeclarations are all used in inconsistent ways so I don't know
if I've tracked the intention of the system correctly.
The PowerPC test change is a bug fix on linux. Previously the manual
conditions were based around !isOSOpenBSD, which is not the condition
where __stack_chk_guard are used. Now getSDagStackGuard returns the
proper global reference, resulting in LOAD_STACK_GUARD getting a
MachineMemOperand which allows scheduling.
We just replaced SmallSet<T *, N> with SmallPtrSet<T *, N>, bypassing
the redirection found in SmallSet.h. With that, we no longer need to
include SmallSet.h in many files.
