A HLASM source file must end with the END instruction. It is implemented
by adding a new function to the target streamer. This change also turns
SystemZHLASMSAsmString.h into a proper header file, and only uses the
SystemZTargetHLASMStreamer when HLASM output is generated.
This PR resolves https://github.com/llvm/llvm-project/issues/144513
The modification include five pattern :
1.vselect Cond, 0, 0 → 0
2.vselect Cond, -1, 0 → bitcast Cond
3.vselect Cond, -1, x → or Cond, x
4.vselect Cond, x, 0 → and Cond, x
5.vselect Cond, 000..., X -> andn Cond, X
1-4 have been migrated to DAGCombine. 5 still in x86 code.
The reason is that you cannot use the andn instruction directly in
DAGCombine, you can only use and+xor, which will introduce optimization
order issues. For example, in the x86 backend, select Cond, 0, x →
(~Cond) & x, the backend will first check whether the cond node of
(~Cond) is a setcc node. If so, it will modify the comparison operator
of the condition.So the x86 backend cannot complete the optimization of
andn.In short, I think it is a better choice to keep the pattern of
vselect Cond, 000..., X instead of and+xor in combineDAG.
For commit, the first is code changes and x86 test(note 1), the second
is tests in other backend(node 2).
---------
Co-authored-by: Simon Pilgrim <llvm-dev@redking.me.uk>
Add handling for FPFastMathMode in SPIR-V shaders. This is a first pass
that
simply does a direct translation when the proper extension is available.
This will unblock work for HLSL. However, it is not a full solution.
The default math mode for spir-v is determined by the API. When
targeting Vulkan many of the fast math options are assumed. We should do
something particular when targeting Vulkan.
We will also need to handle the hlsl "precise" keyword correctly when
FPFastMathMode is not available.
Unblockes https://github.com/llvm/llvm-project/issues/140739, but we are
keeing it open to track the remaining issues mentioned above.
This patch allows urem by a constant to be expanded more efficiently to
avoid the need for expensive udiv instructions. This is part of the
resolution to issue #118090
Consider IR such as this:
for.body:
%iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
%accum = phi i32 [ 0, %entry ], [ %add, %for.body ]
%gep.a = getelementptr i8, ptr %a, i64 %iv
%load.a = load i8, ptr %gep.a, align 1
%ext.a = zext i8 %load.a to i32
%add = add i32 %ext.a, %accum
%iv.next = add i64 %iv, 1
%exitcond.not = icmp eq i64 %iv.next, 1025
br i1 %exitcond.not, label %for.exit, label %for.body
Conceptually we can vectorise this using partial reductions too,
although the current loop vectoriser implementation requires the
accumulation of a multiply. For AArch64 this is easily done with
a udot or sdot with an identity operand, i.e. a vector of (i16 1).
In order to do this I had to teach getScaledReductions that the
accumulated value may come from a unary op, hence there is only
one extension to consider. Similarly, I updated the vplan and
AArch64 TTI cost model to understand the possible unary op.
---------
Co-authored-by: Matt Devereau <matthew.devereau@arm.com>
Code generation predicates like HasSVE2_or_SME implemented a strict
divide between streaming and non-streaming which meant some SME
instructions were not available unless a matching SVE feature was
enabled.
Always try to fold freeze(op(....)) -> op(freeze(),freeze(),freeze(),...).
This patch proposes we drop the opt-in limit for opcodes that are allowed to push a freeze through the op to freeze all its operands, through the tree towards the roots.
I'm struggling to find a strong reason for this limit apart from the DAG freeze handling being immature for so long - as we've improved coverage in canCreateUndefOrPoison/isGuaranteedNotToBeUndefOrPoison it looks like the regressions are not as severe.
Hopefully this will help some of the regression issues in #143102 etc.
This commit fixes using inline assembly with v128 results. Previously
this failed with an internal assertion about a failure to legalize a
`CopyFromReg` where the source register was typed `v8f16`. It looks like
the type used for the destination register was whatever was listed first
in the `def V128 : WebAssemblyRegClass` listing, so the types were
shuffled around to have a default-supported type.
A small test was added as well which failed to generate previously and
should now pass in generation. This test passed on LLVM 18 additionally
and regressed by accident in #93228 which was first included in LLVM 19.
Fixes https://github.com/llvm/llvm-project/issues/50142, a miss of
further vectorization, where we can only achieve zext (xor (any_true),
-1).
Now in test case simd-setcc-reductions, it's converted to all_true.
Also fixes https://github.com/llvm/llvm-project/issues/145177, which is
all_true (setcc x, 0, eq) -> not any_true
any_true (setcc x, 0, ne) -> any_true
all_true (setcc x, 0, ne) -> all_true
---------
Co-authored-by: badumbatish <--show-origin>
[WebAssembly] [Backend] Wasm optimize illegal bitmask for #131980.
Currently, the case for illegal bitmask (v32i8 or v64i8) is that at the
SelectionDag level, two (four) vectors of v128 will be concatenated
together, then they'll all be SETCC by the same pseudo illegal
instruction, which requires expansion later on.
I opt for SETCC-ing them seperately, bitcast and zext them and then add
them up together in the end.
---------
Co-authored-by: badumbatish <--show-origin>
After 901e1390c9778a191256335d37802bc631c2d183 (#127770), the DAG
combine would transform `fma(x, 0.0, 1.0)` into `1.0` if
`-fp-contract=fast` was enabled, in addition to when 'x' is marked
nnan/ninf.
It's only valid in the latter case, not the former, so delete the extra
condition.
This change continues rewriting and cleanup around DAG ISel for
formal-arguments, return values, and function calls. This causes some
incidental changes, mostly to instruction ordering and register naming
but also a couple improvements caused by using scalar types earlier in
the lowering.
These two instructions are supported by gfx1250. We define the
instructions and implement the corresponding intrinsic and builtin.
Co-authored-by: Stanislav Mekhanoshin <Stanislav.Mekhanoshin@amd.com>
Ensure we load BOTH scalars, inserted into different positions into separate vectors with the freeze(poison) base
Noticed while triaging regressions in #145939
When iterating over a block, meta instructions have no effect on wait counts,
but their presence drops the reference to earlier waitcnt instructions before
they are processed. This results in spurious wait counts, which do not affect
correctness, but are also not required in the resulting program. Skipping meta
instructions as soon as they are seen cleans this up.
std::prev(Paired) will get the previous instruction, that might skip
over the instructions in a bundle to the BUNDLE itself. Change it to
Paired->getPrevNode() to make sure we update the registers in each
instruction in the bundle.
This will convert loads of constant strings to immediate values. Put
this behind a flag that is enabled by default so that we can toggle it
if need be.
CTTZ/CTLZ_ZERO_UNDEF nodes can only create poison if the source value is zero - so check with isKnownNeverZero
Pulled out of #146361 and reapplied now that #146490 has landed.
Fixes#141840
This PR implements support for the `memcpy` intrinsic in the DXIL
CBuffer Access pass with the following restrictions:
- The CBuffer Access must be the `src` operand of `memcpy` and must be
direct (i.e., not a GEP)
- The type of the CBuffer Access must be of an Array Type
These restrictions greatly simplify the implementation of `memcpy` yet
still covers the known uses in DML shaders.
Furthermore, to prevent errors like #141840 from occurring silently
again, this PR adds error reporting for unsupported users of globals in
the DXIL CBuffer Access pass.
Remove Linker Optimization Hints (LOHs) from outlining candidates
instead of simply preventing outlining if LOH labels are found in the
candidate. This will improve the effectiveness of the machine outliner
when LOHs are enabled (which is the default).
In
https://discourse.llvm.org/t/loh-conflicting-with-machineoutliner/83279/1
it was observed that the machine outliner is much more effective when
LOHs are disabled. Rather than completely disabling LOH, this PR aims to
keep LOH in most places and removing them from outlined functions where
it could be illegal. Note that we are conservatively removing all LOHs
from outlined functions for simplicity, but I believe we could retain
LOHs that are in the intersection of all candidates.
It should be ok to remove these LOHs since these blocks are being
outlined anyway, which will harm performance much more than the gain
from keeping the LOHs.
It has been observed that the issue width for neoverse-n2 CPUs is set
too high, and does not properly reflect the dispatch constraints.
I tested various values of IssueWidth (10, 8, 6, 5, 4) with runs of
various workloads on a neoverse-n2 machine and I got the highest overall
geomean score with an issue width of 5.
If this patch were to cause any major regression post-commit, it could
be easily reverted, but it is likely to show an overall improvement.
Related Neoverse-V2 PR: https://github.com/llvm/llvm-project/pull/142565
Make the fixed-vector lowering of ISD::[L]LRINT use the custom-lowering
routine, lowerVectorXRINT, and fix issues in lowerVectorXRINT related to
this new functionality.
Neither the arithmetic value or overflow result can create undef/poison from regular operands values.
We have complete test coverage for all ADDO/SUBO nodes, 32-bit codegen handles the _CARRY variants but until #145939 lands AND DAGCombiner::visitFREEZE handles multiple results we can't see any codegen change.
Pulled out of #145939
PR which introduced the bug:
https://github.com/llvm/llvm-project/pull/131837.
Fixes a crash around dead registers which started in f5c62ee0fa04 by
verifying that the reused incoming register is also virtual.
When available, we can use SVE's CNT instruction to improve the lowering
of scalar and fixed-length popcount (CTPOP) since the SVE instruction
supports types that the Neon variant doesn't.
For the scalar types, I see the following speedups on NVIDIA Grace CPU:
| size (bits) | before (Gibit/s) | after (Gibit/s) | speedup |
|------------:|-----------------:|----------------:|--------:|
| 32 | 75.20 | 86.79 | 1.15 |
| 64 | 149.87 | 173.70 | 1.16 |
| 128 | 158.56 | 164.88 | 1.04 |
