We don't need an AND on the last iteration. If we shifted the dividend
due to trailing zeros in the divisor, we don't need a chunk that only
contains shifted in zeros.
If the instruction is part of a bundle, then emitInstruction() will
emit the entire bundle. As such, we should be summing up the sizes
of all instructions in the bundle.
Additionally, do not run the verification if an error has already
occurred. In that case, there may be a size mismatch as a result
of the error.
These came up when trying to enable the verification on additional
targets.
Many backends rely on TII reporting correct instruction sizes for MIR
level branch relaxation passes. Reporting a too small size can result in
MC fixup failures (or silent miscompiles for unvalidated fixups).
Some time ago I added validation to the PPC asm printer to verify that
the TII instruction size matches the actually emitted size. This was
very helpful to systematically fix all incorrectly reported instruction
sizes.
However, the same problem also exists in lots of other backends, so this
moves the validation into AsmPrinter, controlled by a new
getInstSizeVerifyMode() hook in TII, which is disabled by default.
The intention here is to gradually enable this validation for more
backends (which requires fixing them first).
Make the (1 << HBitWidth) % Divisor == 1 path a special case within
the recently added chunk summing algorithm. This allows us to
share the trailing zero shifting code.
While there make some comment improvements and avoid creating
unnecessary nodes.
This replaces LegalVT with HiLoVT and LegalWidth with HBitWidth as
they are the same for all current uses.
Then we rewrite the shifts to operate on LL and LH.
There's a slight regression on RISC-V due to different node creation
order leading to different DAG combine order. I have other refactoring
I'd like to explore then I may try to fix that.
Check the type before we call getOperationAction. Give BuildUDIVPattern
only AllowWiden and a WideSVT.
Update variable names and comments to avoid spreading "64" to too many
places.
Based on the suggestions in #140639, this PR adds the rewrite pattern `a
bitwiseop (~b +/- c)` -> `a bitwiseop ~(b -/+ c)` for AND, OR, and XOR
operations. This rewrite enables lowering to `ANDN`, `ORN`, and `XORN`
operations.
Added new MIR tests in `combine-binop-neg.mir` for AArch64 to verify the
new combine works for various commuted and uncommuted forms with AND,
OR, and XOR and added new LLVM IR tests for RISC-V in `rv32zbb-zbkb.ll`
to ensure the combine produces the expected `ANDN`, `ORN`, and `XORN`
operations.
Selects of the form `cond ? 1 : 0` are created during unrolling of
setcc+vselect. Currently these are not optimized away post-legalization
even if fully redundant. Having these extra selects sitting between
things can prevent other folds from applying.
Enabling this requires some mitigations in the ARM backend, in
particular in the interaction with MVE support. There's two changes
here:
* Form CSINV/CSNEG/CSINC from CMOV, rather than only creating it during
SELECT_CC lowering. (After this change, the lowering in SELECT_CC can be
dropped without test changes, let me know if I should do that.)
* Support pushing negations through CMOV in more cases, in particular if
the operands are constant or the negation can be handled by flipping
lshr/ashr.
Additionally, in the X86 backend, try to simplify CMOV to SETCC if only the
low bit is demanded.
When rematerializing S_MOV_B64 or S_MOV_B64_IMM_PSEUDO and only a single
32-bit lane of the result is used at the remat point, emit S_MOV_B32
with the appropriate half of the 64-bit immediate instead.
This reduces register pressure by defining a 32-bit register instead of
a 64-bit pair when the other half is unused.
This patch adds a Clang-compatible -mtune option to llc, to enable
decoupled ISA and microarchitecture targeting, which is especially
important for backend development. For example, it can enable to easily
test a subtarget feature or scheduling model effects on codegen across a
variaty of workloads on the IR corpus benchmark:
https://github.com/dtcxzyw/llvm-codegen-benchmark.
The implementation adds an isolated generic codegen flag, to establish a
base for wider usage - the plan is to add it to `opt` as well in a
followup patch. Then `llc` consumes it, and sets `tune-cpu` attributes
for functions, which are further consumed by the backend.
The purpose of AtomicExpandUtils.h was to allow code reuse with [PNaCl
](https://en.wikipedia.org/wiki/Google_Native_Client)which has since
been deprecated. This PR removes the file, and migrates the declarations
to be visible only in the necessary scopes.
FastISel was dropping llvm.fake.use because they are not meant to be
generated at O0 with clang.
This patch adds support in FastISel to generate FAKE_USE for llvm.fake.use.
The handling is simpler than in SelectionDagBuilder because no attempt is made to
get rid of useless FAKE_USE (e.g. for constant SSA values) to keep FastISel simple.
The motivation is that flang will generate llvm.fake.use for function arguments under
`-g` (and O0) because Fortran arguments are not copied to the stack (they are
reference like arguments in most cases) and one should be able to access these
variables from the debugger at any point of the function, even after their last use in the
function.
This patch improves the lowering of 128-bit unsigned division and
remainder by constants (UDIV/UREM) by avoiding a fallback to libcall
(__udivti3/uremti3) for specific divisors.
When a divisor D satisfies the condition (1 << ChunkWidth) % D == 1, the
128-bit value is split into fixed-width chunks (e.g., 30-bit) and summed
before applying a smaller UDIV/UREM. This transformation is based on the
"remainder by summing digits" trick described in Hacker’s Delight.
This fixes#137514 for some constants.
The index operand of ISD::EXTRACT_SUBVECTOR is implicitly scaled by
vscale, which is effectively always one for fixed-length vectors. When
combining nested extracts we must ensure all use the same implicit
scaling otherwise the transform is not equivalent.
Fixes https://github.com/llvm/llvm-project/issues/186563
Replace the DenseMap from blocks to their innermost loop a vector
indexed by block numbers, when possible. Supporting number updates is
not trivial as we don't store a list of basic blocks, so this is not
implemented.
NB: I'm generally not happy with the way loops are stored. As I think
that there's room for improvement, I don't want to touch the
representation at this point.
Pull Request: https://github.com/llvm/llvm-project/pull/103400
Guard against null DISubroutineType when checking for variadic
parameters in `constructSubprogramScopeDIE`. `DISubprograms` may lack a
type field when using LineTablesOnly emission, causing a null pointer
dereference.
Fixes#184003
Co-authored-by: Shivam Kunwar <phyBrackets@users.noreply.github.com>
This moves it above the type legality check. The legality check we use
for the main division by constant algorithm is probably not right for
BuildExactSDIV and BuildExactSDIV. These checks are largely about the
legality of MUL_LOHI/MULH which are not used for the exact case.
This patch removes the legal type check for the exact case. If we do
need a check it's probably better to have a specific version in
BuildExactSDIV and BuildExactSDIV.
I'm hoping to do some refactoring of the legality checks in
BuildSDIV/BuildUDIV so separating them makes this easier.
Remarks sections are no longer necessary for basic functioning of
bitstream remarks, so only fire a warning if remarks sections are
force-enabled.
Pull Request: https://github.com/llvm/llvm-project/pull/187362
Usually `dump()`s are without parameter, so the practice is calling
`XXX::dump()` when debugging.
But we will get an error like below after #161097:
```
error: <user expression 128>:1:10: too few arguments to function call,
expected 1, have 0
1 | DAG.dump()
| ~~~~~~~~ ^
```
So to not surprise users, I added back the `SelectionDAG::dump()`
without parameter.
Adds support for `atomicrmw` `fminimumnum`/`fmaximumnum` operations.
These were added to C++ in P3008, and are exposed in libc++ in #186716 .
Adding LLVM IR support for these unblocks work in both backends with HW
support, and frontends.
Alternative approach to the same goals as #162407
This takes `TargetLoweringBase::getLoadExtAction`, renames it to
`TargetLoweringBase::getLoadAction`, merges `getAtomicLoadExtAction`
into it, and adds more inputs for relavent information (alignment,
address space).
The `isLoadExtLegal[OrCustom]` helpers are also modified in a matching
manner.
This is fully backwards compatible, with the existing `setLoadExtAction`
working as before. But this allows targets to override a new hook to
allow the query to make more use of the information. The hook
`getCustomLoadAction` is called with all the parameters whenever the
table lookup yields `LegalizeAction::Custom`, and can return any other
action it wants.
Block numbers are updated too frequently, which makes it difficult to
keep analyses up to date. Therefore, introduce a second number per basic
block that is used for analyses and is renumbered less often. This frees
analyses from providing somewhat efficient facilities for dealing with
changed block numbers, making it simpler to implement in e.g. LoopInfo
or CycleInfo.
(Currently, "less often" means not at all, but we might want to renumber
after certain passes if the numbering gets too sparse and no analyses
are preserved anyway.)
When we introduced a more general use of block numbers some time ago,
using the existing numbers seemed to be a somewhat obvious choice, but I
now think that this was a bad decision, as it conflates a number that is
used for ordering with a number that should be more stable.
MachineBasicBlock isn't particularly size-optimized and there's a fair
amount of padding where we can add another number.
There should be no performance impact,
This patch relands https://github.com/llvm/llvm-project/pull/178666. The
original version caused CI failures due to the missing target triple in
`llvm/test/CodeGen/X86/byte-constants.ll`. CI should be green now.
Currently llvm.experimental.cttz.elts are directly lowered from the
intrinsic.
If the type isn't legal then the target tells SelectionDAGBuilder to
expand it into a reduction, but this means we can't split the operation.
E.g. it's possible to split a cttz.elts nxv32i1 into two nxv16i1,
instead of expanding it into a nxv32i64 reduction.
vp.cttz.elts can be split because it has a dedicated SelectionDAG node.
This adds CTTZ_ELTS and CTTZ_ELTS[_ZERO_POISON] nodes and just enough
legalization to get tests passing. A follow up patch will add splitting
and move the expansion into LegalizeDAG.
BranchProbabilityInfo will compute it's own dominator tree and
post-dominator tree if none is specified; avoid this by using the
analysis manager/pass manager to get the analysis, which will reuse the
previously computed DomTree.
Fixes#181651
Added DemandedElts argument to isConstOrConstSplat and to
isKnowTobePowerOfTwo calls and OrZero || isKnownNeverZero(Val, Depth) is
checked before isKnowTobePowerOfTwo. Also added unit tests.
Largely a straight-forward replacement with occasional simplifcations.
For AMDGPU, I assumed that unconditional branches are always uniform and
therefore "simplified"/changed AMDGPUAnnotateUniformValues to only
annotate conditional branches.
Target-specific FastISel only selects conditional branches,
unconditional branches are already handled by the non-target-specific
code.
Unlike LLVM IR `Instruction::eraseFromParent()`,
`MachineInstr::eraseFromParent()` is void and does not return the
iterator following the erased instruction. Returning an iterator can be
very helpful for example when we are erasing MachineInstrs while
iterating, as it provides a convenient way to get a valid iterator.
This patch updates `MachineInstr::eraseFromParent()` to return a
`MachineBlock::iterator` (which is a
`MachineInstrBundleIterator<MachineInstr>`). If the erased instruction
is the head of a bundle, then the returned iterator points to the next
bundle (see unittest).
Support parsing and printing inline assembly operands in MIR using the
symbolic form instead of numeric register class IDs, thus removing the
need to update tests when the numbers change.
The numeric form remains supported.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This patch changes the lowering of the [experimental.memset.pattern intrinsic](https://llvm.org/docs/LangRef.html#llvm-experimental-memset-pattern-intrinsic)
to match the optimized memset and memcpy lowering when possible. (The tl;dr of
memset.pattern is that it is like memset, except that you can use it to set
values that are wider than a single byte.)
The memset.pattern lowering now queries `TTI::getMemcpyLoopLoweringType` for a
preferred memory access type. If the size of that type is a multiple of the set
value's type, and if both types have consistent store and alloc sizes (since
memset.pattern behaves in a way that is not well suitable for access widening
if store and alloc size differ), the memset.pattern is lowered into two loops:
a main loop that stores a sufficiently wide vector splat of the SetValue with
the preferred memory access type and a residual loop that covers the remaining
set values individually.
In contrast to the memset lowering, this patch doesn't include a specialized
lowering for residual loops with known constant lengths. Loops that are
statically known to be unreachable will not be emitted.
For backends that don't override `TTI::getMemcpyLoopLoweringType`, the
generated code is mostly unchanged except for more consistent basic block
names, no more `br i1 false` for memset.patterns with known size, and a flipped
loop condition for memset.patterns with known size (see test changes).
This is a follow-up to a similar patch for memset: #169040
