The original implementation performed the transformation when
isTruncateFree was true:
truncate(build_vector(x, x)) -> build_vector(truncate(x), truncate(x)).
In some cases, x comes from an ext, try to pre-truncate build_vectors
source operands
when the source operands of build_vectors comes from an ext.
Testcase from: https://gcc.godbolt.org/z/bbxbYK7dh
This change landed without approval.
This reverts commit 45e666a8531c1148bdb170b9a120f99e1500c427.
This reverts commit a636dd4c37f12594275de2fe180ca35bc04d76ea.
This patch fixes an assertion failure in ForwardStoreValueToDirectLoad
during DAGCombine.
The crash occurs when `STLF (Store-to-Load Forwarding)` creates an
illegal intermediate bitcast type (e.g., `v128i1` when bridging a
128-bit store to a `<32 x i1>` load on X86). Since `v128i1` is not a
legal mask type for the backend, it violates the expectations of the
LegalizeDAG pass.
The fix adds a `TLI.isTypeLegal(InterVT)` check to ensure that the
intermediate type used for the transformation is supported by the
target.
Fixes#181130
Clang isn't able to support multiple constraints on inputs and outputs,
like "rm". Instead, it picks the "safest" one to use, i.e. the memory
constraint for "rm". This leads to obviously horrible code:
asm __volatile__ ("pushf\n\t"
"popq %0"
: "=rm" (x));
is compiled to:
pushf
popq -8(%rsp)
movq -8(%rsp), %rax
It gets worse when inlined into other functions, because it may
introduce
a stack where none is needed.
With this change, Clang now generates IR for the more optimistic choice
("r"). All but the fast register allocator are able to fold registers if
it turns out that register pressure is too high.
This leaves the fast register allocator. The fast register allocator, as
the name suggests, is built for execution speed, not code quality. Thus,
we add special processing to convert the "optimistic" IR into the
"conservative" choice (again at the IR level), which we know it can
handle.
We focus on "rm" for the initial commit, but that can be expanded in the
future for other constraints where Clang generates ++ungood code (like
"g").
Fixes: https://github.com/llvm/llvm-project/issues/20571
### Summary
Fix type mismatch in ternary expression that causes GCC `-Werror=extra`
to fail.
### Details
GCC's `-Werror=extra` enforces stricter type consistency in ternary
expressions, in this case unsigned and an enum literal.
### Tested
- Built with ToT clang and GCC 13.3.0 on Linux x86_64 (not really
because there are other warnings, but this one is gone).
- All existing tests pass
The '``llvm.looptrap``' intrinsic is equivalent to
``llvm.cond.loop(true)``, but is also considered to be ``noreturn``,
which enables certain optimizations by allowing the optimizer to
assume that a branch leading to a call to this intrinsic was not
taken. A late optimization pass will convert this intrinsic to either
``llvm.cond.loop(true)`` or ``llvm.cond.loop(pred)``, where ``pred``
is a predicate for a conditional branch leading to the intrinsic call,
if possible.
Reviewers: fmayer, vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/181299
Instead of trying to make CTLS work for the Lo part, conditionally
invert Lo using the Hi sign bit, then do a CTLZ. If the CTLZ is
zero, then the Lo sign bit differs from the Hi sign bit. Otherwise,
each leading zero represents an additional sign bit.
This generates better code when CTLS and CTLZ are both supported.
I've added Zbb to the P extension command line for RISC-V since
P is likely to imply Zbb when it is ratified, but that isn't written
in the spec yet. If that doesn't happen, I expect CLZ would get
added back to the P extension.
Currently, the compiler doesn't create a COPY for undef operands while
lowering REG_SEQUENCE, and only if LIS information is available, it
propagates the undef flag to the subreg uses. So, if LIS isn't
available, we can end up with some uses without def of those lanes.
Now, we check which lanes are used in a single scan of
use_nodbg_operands() per REG_SEQ, and perform the skip of the COPY only
if LIS is avaible (as undef will be propagated later) or if there are no
uses for that lane.
There is still a scan of the use list, but now it's only one per REG_SEQ
and I think it's necessary, as there is no guarantee to have LIS or
other analysis pass information at this stage.
This is a proposal fix for issue:
https://github.com/llvm/llvm-project/issues/175596
---------
Co-authored-by: Carl Ritson <critson@perlfu.co.uk>
DAGCombiner can fold a chain of INSERT_VECTOR_ELT into a vector AND/OR
operation. This patch adds protection to avoid that we end up making the
vector more poisonous by freezing the source vector when the elements
that should be set to 0/-1 may be poison in the source vector.
The patch also fixes a bug in SimplifyDemandedVectorElts for
MUL/MULHU/MULHS/AND that could result in making the vector more
poisonous. Problem was that we skipped demanding elements from Op0 that
were known to be zero in Op1. But that could result in elements being
simplified into poison when simplifying Op0, and then the result would
be poison and not zero after the MUL/MULHU/MULHS/AND. The solution is to
defensively make sure that we demand all the elements originally
demanded also when simplifying Op0.
This bugs were found when analysing the miscompiles in
https://github.com/llvm/llvm-project/issues/179448
Main culprit in #179448 seems to have been the bug in DAGCombiner. The
bug in SimplifyDemandedVectorElts surfaced when fixing the DAGCombiner,
as that fix typically introduce the (AND (FREEZE x), y) pattern that
wasn't handled correctly in SimplifyDemandedVectorElts.
Also fixes#180409.
Also fixes#176682.
This PR fixes a big-endian regression in `ForwardStoreValueToDirectLoad`
where the wrong subvector was being extracted. In big-endian, memory
offset 0 corresponds to the high bits, so the extraction index needs to
be adjusted.
As suggested by @KennethHilmersson, calculate the extraction index as
the difference between the number of elements in the intermediate vector
and the load vector when in big-endian mode.
Special thanks to Kenneth Hilmersson for providing the fix logic and the
ARM regression test.
https://github.com/llvm/llvm-project/pull/172523#issuecomment-3878065191https://github.com/llvm/llvm-project/pull/172523#issuecomment-3879575092
I plan to use that to optimize mask creation in VPlan predicator by
`or`ing edge masks from the post-dominance frontier instead of all
predecessors in a subsequent patch. Note that it would require to use
the same unmodified post-dom tree for *all* the basic blocks in a VPlan
that is already limited to a particular loopnest so the algorithmic
complexity concerns behind the "deprecation" notice in the beggining of
`DominanceFrontier.h` (and also discussion in the
https://discourse.llvm.org/t/dominance-frontiers/21755 thread) don't
apply for my use case (at least to the best of my understanding).
The change here is to properly use graph-traits for children traversal
plus inline `ForwardDominanceFrontierBase` into `DominanceFrontierBase` now
that it's used for post-dom-frontier.
Since the only planned use-case is in the vectorizer, I'm adding a
VPlan-base unittest along with this change.
Support for multiple root nodes via using virtual root node to seed the
worklist is left to a subsequent PR.
Add support for expanding fptosi.sat and fptoui.sat via IR expansions.
Similar to fptosi/fptoui we would get legalization errors otherwise.
The previous expansion for fptosi/fptoui was already saturating -- but
those instructions do not actually require saturation, and the
implementation of the saturation was incorrect in lots of ways. What
this PR does is:
* For fptosi, remove the unnecessary saturation handling.
* For fptoui, remove the unnecessary saturation handling and sign
multiplication.
* For fptosi, use the previous saturation handling with fixes: We need
to map NaNs to 0 and the saturation condition on the exponent was
incorrect. (I'm performing the NaN check via fcmp -- there's no
requirement to do everything bitwise here.)
* For fptoui use a variation of the signed saturation handling: Negative
values need to go to zero and we saturate to unsigned max.
Proofs: https://alive2.llvm.org/ce/z/Xv9FNd
MIPS16 cannot handle constant pools created by CTTZ table lookup
expansion. This causes "Cannot select" errors when trying to select
MipsISD::Lo nodes for constant pool addresses.
Modify the table lookup conditions to check ConstantPool operation
status, and only set ConstantPool to Custom in non-MIPS16 mode in MIPS
backend.
This ensures MIPS16 uses the ISD::CTPOP instead of attempting
unsupported constant pool operations.
Fix#61055.
This change turns the `"nooutline"` attribute into an enum attribute
called `nooutline`, and adds an auto-upgrader for bitcode to make the
same change to existing IR.
This IR attribute disables both the Machine Outliner (enabled at Oz for
some targets), and the IR Outliner (disabled by default).
Two code paths in `reassociationCanBreakAddressingModePattern` were
missing a `hasUniqueMemOperand()` guard before calling
`getAddressSpace()`. Note that on `L1214` we already have the same guard
in place.
`getAddressSpace()` chains through `getPointerInfo()` to
`getMemOperand()`, which asserts that the node has exactly one memory
operand.
This lowers the splitting as:
```
any_active(hi_mask)
? (find_last_active(hi_mask) + lo_mask.getVectorElementCount())
: find_last_active(lo_mask)
```
And trivially lowers `<1 x i1>` scalarization to returning zero. Which
is a natural result of the splitting (and the lack of a sentinel
"none-active" result value).
The lowerings likely can be improved. This patch is for completeness.
Should fix:
https://github.com/llvm/llvm-project/pull/178862#issuecomment-3862310334Fixes#180212
When there is no target-specific lowering of @llvm.cond.loop, it is
lowered into a simple loop by PreISelIntrinsicLowering. Mark the branch
weights into the no-return loop as unknown given we do not have value
metadata to fix the profcheck test for this feature.
Reviewers: mtrofin, alanzhao1, snehasish, pcc
Pull Request: https://github.com/llvm/llvm-project/pull/180390
Adresses one of the subtasks of #150515.
The code is ported from `SelectionDAG::computeKnownBits` and tests are
loosely based on `AArch64/GlobalISel/knownbits-shl.mir`.
The llvm.cond.loop intrinsic is semantically equivalent to a conditional
branch conditioned on ``pred`` to a basic block consisting only of an
unconditional branch to itself. Unlike such a branch, it is guaranteed
to use specific instructions. This allows an interrupt handler or
other introspection mechanism to straightforwardly detect whether
the program is currently spinning in the infinite loop and possibly
terminate the program if so. The intent is that this intrinsic may
be used as a more efficient alternative to a conditional branch to
a call to ``llvm.trap`` in circumstances where the loop detection
is guaranteed to be present. This construct has been experimentally
determined to be executed more efficiently (when the branch is not taken)
than a conditional branch to a trap instruction on AMD and older Intel
microarchitectures, and is also more code size efficient by avoiding the
need to emit a trap instruction and possibly a long branch instruction.
On i386 and x86_64, the infinite loop is guaranteed to consist of a short
conditional branch instruction that branches to itself. Specifically,
the first byte of the instruction will be between 0x70 and 0x7F, and
the second byte will be 0xFE.
Part of this RFC:
https://discourse.llvm.org/t/rfc-optimizing-conditional-traps/89456
Reviewers: arsenm, RKSimon, fmayer, vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/177686
Profile data for instructions (e.g., branch weights) is automatically
preserved via `splice()` which moves the basic blocks along with their
instruction metadata. However, entry count is stored as function
metadata, which was dropped when creating merged function and thunks.
The fix is to explicitly set entry count for both merged function (.Tgm)
and thunks via `setEntryCount()`.
This patch fixes a regression introduced by PR #175022, where
a freeze was introduced with the following transformation:
ext(freeze(load(x))) -> freeze(extload(x))
If a new extend is introduced afterwards we then have
ext(freeze(extload(x)))
which doesn't get picked up by existing DAG combines due to
the freeze getting in the way.
Previously, the DAG combiner did not optimize exact signed division by a
power-of-two constant divisor for integer types exceeding the size of
division supported by the target architecture (e.g., i128 on x86-64).
However, such an optimization was expected by the division expansion
logic, leading to unsupported division operations making it to
instruction selection.
This commit addresses this issue by making an exception to the existing
exclusion of signed division with the exact flag for the aforementioned
operations. That is, the DAG combiner will now optimize exact signed
division if the divisor is a power-of-two constant and the integer type
exceeds the size of division supported by the target architecture.
---------
Signed-off-by: Steffen Holst Larsen <HolstLarsen.Steffen@amd.com>
Handle this case by extending the integer to a wider type. This can
probably be handled more optimally, but this is conservatively correct.
Proof: https://alive2.llvm.org/ce/z/0RwDO1
