The very first step towards #83422 - which will move DAG combines to be
processed in topological order.
There is a lot of churn on existing tests that need to be addressed
before this can be switched on globally, this patch gives the ability to
enable it both on a per-target basis, and via a command line option to
assist with testing and triage.
At the moment I'm focusing on addressing the x86 regressions (example in
the patch's basic test coverage) as that's the target I'm most familiar
with and will help with many other targets as well, but there might be
other/simpler targets that would benefit from earlier handling.
After #187378 these are no longer tested. I'm concerned that we can
create illegal scalar types after type legalization. I don't know how to
test this now so I'd like to remove support until it is needed and can
be tested.
If we we are going to legalize to a vector with the same element type
and mulh or mul_lohi are supported, allow the optimization before type
legalization.
RISC-V will widen vectors using vp.udiv/sdiv that doesn't support
division by constant optimization. In addition, type legalization will create
a build_vector with undef elements making it hard to match after type
legalization.
Other targets may need to widen by a combination of vector and scalar
divisions to avoid traps if we widen a vector with garbage.
I had to enable the MULHU->SRL DAG combine before type legalization to
prevent regressions. After type legalization, the multiply constant
build_vector will have undef elements and the combine won't trigger.
K is an unsigned, it will be zero extended to uint64_t for
the APInt constructor. If the ShSVT has more than 32 bits, we won't
create an all ones ConstantSDNode.
To fix this, explicitly push an all ones constant to KAmts. This
also fixes an APInt ImplicitTrunc.
This allows turnVectorIntoSplatVector to work for this case.
Resolves#175150
Defines computeConstantRange and computeConstantRangeIncludingKnownBits
in the SelectionDAG. Currently only handles `ISD::VSCALE` operation
related to #174708.
Test cases were constructed to test varying VSCALE ranges on AArch64.
Further testing can be implemented as needed by review.
The -1 has 'int' type. The APInt assignment operator takes uint64_t.
Fortunately, due to C rules, the -1 will be converted to an all ones
uint64_t. Unfortunately, if the APInt has more than 64 bits, the upper
words will be zeroed. I don't think we have any testing of that today.
Use setAllBits to avoid the subtle cast and fix the bits > 64 issue.
K still has its own issue that needs to be fixed.
Before handling `VECTOR_FIND_LAST_ACTIVE` expansion, ensure the vector
mask elements get extended accordingly, after the vector mask type is
promoted to its legal type.
Fixes: https://github.com/llvm/llvm-project/issues/187875.
This reduces the number of generated instructions in some cases.
While I was there, I changed an Add to a Disjoint OR since DAGCombine
was going to do it anyway.
Currently a cttz.elts of e.g. nxv32i1 will get expanded to a reduction
of nxv32i64 or equivalent, but we can split it into two legal nxv16i1
cttz.elts once we have dedicated SelectionDAG nodes.
This implements the splitting for them the same way we implement type
splitting for vp.cttz.elts, i.e. check if the low result is VF, and if
so add it to the result of the high result. It also implements operand
type promotion for NEON which needs to promote i1 vectors to something
larger first.
We also need to move expansion into LegalizeVectorOps so it doesn't get
expanded before type legalization can do splitting. This uses
LegalizeVectorOps in case the scalar reduction type, which depends on
the minimum bitwidth needed to store the result, still needs type
promotion.
The TTI costs should be updated after this to reflect the more efficient
codegen, but that is deferred to another PR.
When legalizing CTLZ for a wider type with a SUB bit-width compensation,
the result for a zero-defined case can have an extra bit that exceeds
the maximum value of the original type width. Unlike CTLS, which covers
all bits within its type width, CTLZ leads to producing undefined
KnownBits. So, after subtraction, the KnownBits could represent a
negative value, which prevents subsequent optimizations.
This patch ensures that non-negative information is preserved during the
SUB(CTLZ) expansion, so that KnownBits after SUB will take a stricter
range within the type width, thus allowing further combine optimizations
following by SUB. Particularly, this enables the DAG Combiner to remove
unnecessary AND instructions for CTLZ output type conversion.
Fixes#136516
---------
Co-authored-by: Simon Pilgrim <git@redking.me.uk>
This patch change the type of the step vector lowered from
`expandVectorFindLastActive` from `e8` to the index type of the target
machine.
This can help the index out of bound issue when the VLEN is large.
Note that after this patch, there are still some issue in
expandVectorFindLastActive.
We don't need an AND on the last iteration. If we shifted the dividend
due to trailing zeros in the divisor, we don't need a chunk that only
contains shifted in zeros.
Make the (1 << HBitWidth) % Divisor == 1 path a special case within
the recently added chunk summing algorithm. This allows us to
share the trailing zero shifting code.
While there make some comment improvements and avoid creating
unnecessary nodes.
This replaces LegalVT with HiLoVT and LegalWidth with HBitWidth as
they are the same for all current uses.
Then we rewrite the shifts to operate on LL and LH.
There's a slight regression on RISC-V due to different node creation
order leading to different DAG combine order. I have other refactoring
I'd like to explore then I may try to fix that.
Check the type before we call getOperationAction. Give BuildUDIVPattern
only AllowWiden and a WideSVT.
Update variable names and comments to avoid spreading "64" to too many
places.
Selects of the form `cond ? 1 : 0` are created during unrolling of
setcc+vselect. Currently these are not optimized away post-legalization
even if fully redundant. Having these extra selects sitting between
things can prevent other folds from applying.
Enabling this requires some mitigations in the ARM backend, in
particular in the interaction with MVE support. There's two changes
here:
* Form CSINV/CSNEG/CSINC from CMOV, rather than only creating it during
SELECT_CC lowering. (After this change, the lowering in SELECT_CC can be
dropped without test changes, let me know if I should do that.)
* Support pushing negations through CMOV in more cases, in particular if
the operands are constant or the negation can be handled by flipping
lshr/ashr.
Additionally, in the X86 backend, try to simplify CMOV to SETCC if only the
low bit is demanded.
FastISel was dropping llvm.fake.use because they are not meant to be
generated at O0 with clang.
This patch adds support in FastISel to generate FAKE_USE for llvm.fake.use.
The handling is simpler than in SelectionDagBuilder because no attempt is made to
get rid of useless FAKE_USE (e.g. for constant SSA values) to keep FastISel simple.
The motivation is that flang will generate llvm.fake.use for function arguments under
`-g` (and O0) because Fortran arguments are not copied to the stack (they are
reference like arguments in most cases) and one should be able to access these
variables from the debugger at any point of the function, even after their last use in the
function.
This patch improves the lowering of 128-bit unsigned division and
remainder by constants (UDIV/UREM) by avoiding a fallback to libcall
(__udivti3/uremti3) for specific divisors.
When a divisor D satisfies the condition (1 << ChunkWidth) % D == 1, the
128-bit value is split into fixed-width chunks (e.g., 30-bit) and summed
before applying a smaller UDIV/UREM. This transformation is based on the
"remainder by summing digits" trick described in Hacker’s Delight.
This fixes#137514 for some constants.
The index operand of ISD::EXTRACT_SUBVECTOR is implicitly scaled by
vscale, which is effectively always one for fixed-length vectors. When
combining nested extracts we must ensure all use the same implicit
scaling otherwise the transform is not equivalent.
Fixes https://github.com/llvm/llvm-project/issues/186563
This moves it above the type legality check. The legality check we use
for the main division by constant algorithm is probably not right for
BuildExactSDIV and BuildExactSDIV. These checks are largely about the
legality of MUL_LOHI/MULH which are not used for the exact case.
This patch removes the legal type check for the exact case. If we do
need a check it's probably better to have a specific version in
BuildExactSDIV and BuildExactSDIV.
I'm hoping to do some refactoring of the legality checks in
BuildSDIV/BuildUDIV so separating them makes this easier.
Usually `dump()`s are without parameter, so the practice is calling
`XXX::dump()` when debugging.
But we will get an error like below after #161097:
```
error: <user expression 128>:1:10: too few arguments to function call,
expected 1, have 0
1 | DAG.dump()
| ~~~~~~~~ ^
```
So to not surprise users, I added back the `SelectionDAG::dump()`
without parameter.
Adds support for `atomicrmw` `fminimumnum`/`fmaximumnum` operations.
These were added to C++ in P3008, and are exposed in libc++ in #186716 .
Adding LLVM IR support for these unblocks work in both backends with HW
support, and frontends.
Alternative approach to the same goals as #162407
This takes `TargetLoweringBase::getLoadExtAction`, renames it to
`TargetLoweringBase::getLoadAction`, merges `getAtomicLoadExtAction`
into it, and adds more inputs for relavent information (alignment,
address space).
The `isLoadExtLegal[OrCustom]` helpers are also modified in a matching
manner.
This is fully backwards compatible, with the existing `setLoadExtAction`
working as before. But this allows targets to override a new hook to
allow the query to make more use of the information. The hook
`getCustomLoadAction` is called with all the parameters whenever the
table lookup yields `LegalizeAction::Custom`, and can return any other
action it wants.
This patch relands https://github.com/llvm/llvm-project/pull/178666. The
original version caused CI failures due to the missing target triple in
`llvm/test/CodeGen/X86/byte-constants.ll`. CI should be green now.
Currently llvm.experimental.cttz.elts are directly lowered from the
intrinsic.
If the type isn't legal then the target tells SelectionDAGBuilder to
expand it into a reduction, but this means we can't split the operation.
E.g. it's possible to split a cttz.elts nxv32i1 into two nxv16i1,
instead of expanding it into a nxv32i64 reduction.
vp.cttz.elts can be split because it has a dedicated SelectionDAG node.
This adds CTTZ_ELTS and CTTZ_ELTS[_ZERO_POISON] nodes and just enough
legalization to get tests passing. A follow up patch will add splitting
and move the expansion into LegalizeDAG.
Fixes#181651
Added DemandedElts argument to isConstOrConstSplat and to
isKnowTobePowerOfTwo calls and OrZero || isKnownNeverZero(Val, Depth) is
checked before isKnowTobePowerOfTwo. Also added unit tests.
Largely a straight-forward replacement with occasional simplifcations.
For AMDGPU, I assumed that unconditional branches are always uniform and
therefore "simplified"/changed AMDGPUAnnotateUniformValues to only
annotate conditional branches.
Target-specific FastISel only selects conditional branches,
unconditional branches are already handled by the non-target-specific
code.
BranchInst currently represents both unconditional and conditional
branches. However, these are quite different operations that are often
handled separately. Therefore, split them into separate opcodes and
classes to allow distinguishing these operations in the type system.
Additionally, this also slightly improves compile-time performance.
This is similar to the FSHL/FSHR handling in
hoistLogicOpWithSameOpcodeHands.
Here the opcodes aren't exactly the same, but the operations are
equivalent.
Fixes regressions from #180888
