Disable fixed-point iteration in all AMDGPU Combiners after #102163.
This saves around 2% compile time in ad hoc testing on some large
graphics shaders. I did not notice any regressions in the generated
code, just a bunch of harmless differences in instruction selection and
register allocation.
This mirrors what GISel already does, extending the existing lowering of
aarch64_neon_saddlv/aarch64_neon_uaddlv to SADDLV/UADDLV. This allows us
to remove some tablegen patterns, and provides a little nicer codegen in
places as the nodes represent the result being in a vector register
correctly.
This allows us to handle return values that are too large to fit in x10
and x11. They will be converted to a sret by passing a pointer to where
to store the return value.
This is a fairly narrow transform (at the moment) to reduce the VLs of
instructions feeding a store with a smaller VL. Note that the goal of
this transform isn't really to reduce VL - it's to reduce VL *toggles*.
To our knowledge, small reductions in VL without also changing LMUL are
generally not profitable on existing hardware.
For a single use instruction without side effects, fp exceptions, or a
result dependency on VL, reducing VL is legal if only a subset of
elements are legal. We'd already implemented this logic for vmv.v.v, and
this patch simply applies it to stores as an alternate root.
Longer term, I plan to extend this to other root instructions (i.e.
different kind of stores, reduces, etc..), and add a more general
recursive walkback through operands.
One risk with the dataflow based approach is that we could be reducing
VL of an instruction scheduled in a region with the wider VL (i.e. mixed
mode computations) forcing an additional VL toggle. An example of this
is the @insert_subvector_dag_loop test case, but it doesn't appear to
happen widely. I think this is a risk we should accept.
This allows the use a single wider operation with a restricted EVL
instead of padding the vector with the neutral element.
For RISCV specifically, it's worth noting that an alternate padded
lowering is available when VL is one less than a power of two, and LMUL
<= m1. We could slide the vector operand up by one, and insert the
padding via a vslide1up. We don't currently pattern match this, but we
could. This form would arguably be better iff the surrounding code
wanted VL=4. This patch will force a VL toggle in that case instead.
Basically, it comes down to a question of whether we think odd sized
vectors are going to appear clustered with odd size vector operations,
or mixed in with larger power of two operations.
Note there is a potential downside of using vp nodes; we loose any
generic DAG combines which might have applied to the widened form.
These builtins are currently returning CR0 which will have the format
[0, 0, flag_true_if_saved, XER].
We only want to return flag_true_if_saved. This patch adds a shift to
remove the XER bit before returning.
Currently targets without LZCNT/TZCNT won't speculate with BSR/BSF instructions in case they have a zero value input, meaning we always insert a test+branch for the zero-input case.
This patch proposes we allow speculation if the target has CMOV, and perform a branchless select instead to handle the zero input case. This will predominately help x86-64 targets where we haven't set any particular cpu target. We already always perform BSR/BSF instructions if we were lowering a CTLZ/CTTZ_ZERO_UNDEF instruction.
Implement lowering of the Mul/Div operations and also shift parts
operations. Implement lowering of the bit manipulations, like
ROT/SWAP/CTPOP/CTTZ/CTLZ.
This PR continues https://github.com/llvm/llvm-project/pull/101732
changes in virtual register processing aimed to improve correctness of
emitted MIR between passes from the perspective of MachineVerifier.
Namely, the following changes are introduced:
* register classes (lib/Target/SPIRV/SPIRVRegisterInfo.td) and
instruction patterns (lib/Target/SPIRV/SPIRVInstrInfo.td) are corrected
and simplified (by removing unnecessary sophisticated options) -- e.g.,
this PR gets rid of duplicating 32/64 bits patterns, removes ANYID
register class and simplifies definition of the rest of register
classes,
* hardcoded LLT scalar types in passes before instruction selection are
corrected -- the goal is to have correct bit width before instruction
selection, and use 64 bits registers for pattern matching in the
instruction selection pass; 32-bit registers remain where they are
described in such terms by SPIR-V specification (like, for example,
creation of virtual registers for scope/mem semantics operands),
* rework virtual register type/class assignment for calls/builtins
lowering,
* a series of minor changes to fix validity of emitted code between
passes:
- ensure that that bitcast changes the type,
- fix the pattern for instruction selection for OpExtInst,
- simplify inline asm operands usage,
- account for arbitrary integer sizes / update legalizer rules;
* add '-verify-machineinstrs' to existed test cases.
See also https://github.com/llvm/llvm-project/issues/88129 that this PR
may resolve.
This PR fixes a great number of issues reported by MachineVerifier and,
as a result, reduces a number of failed test cases for the mode with
expensive checks set on from ~200 to ~57.
CreateControlFlowHub is a method that redirects control flow edges from a set of
incoming blocks to a set of outgoing blocks through a new set of "guard" blocks.
This is now refactored into a separate file with one enhancement: The input to
the method is now a set of branches rather than two sets of blocks.
The original implementation reroutes every edge from incoming blocks to outgoing
blocks. But it is possible that for some incoming block InBB, some successor S
might be in the set of outgoing blocks, but that particular edge should not be
rerouted. The new implementation makes this possible by allowing the user to
specify the targets of each branch that need to be rerouted.
This is needed when improving the implementation of FixIrreducible #101386.
Current use in FixIrreducible does not demonstrate this finer control over the
edges being rerouted. But in UnifyLoopExits, when only one successor of an
exiting block is an exit block, this refinement now reroutes only the relevant
control-flow through the edge; the non-exit successor is not rerouted. This
results in fewer branches and PHI nodes in the hub.
The liveins were always for an LMUL=1 register class even if the
first instruction used a larger regsister class.
One test in zext.mir used the wrong class for the first instruction.
This commit introduces attribute bpf_fastcall to declare BPF functions
that do not clobber some of the caller saved registers (R0-R5).
The idea is to generate the code complying with generic BPF ABI,
but allow compatible Linux Kernel to remove unnecessary spills and
fills of non-scratched registers (given some compiler assistance).
For such functions do register allocation as-if caller saved registers
are not clobbered, but later wrap the calls with spill and fill
patterns that are simple to recognize in kernel.
For example for the following C code:
#define __bpf_fastcall __attribute__((bpf_fastcall))
void bar(void) __bpf_fastcall;
void buz(long i, long j, long k);
void foo(long i, long j, long k) {
bar();
buz(i, j, k);
}
First allocate registers as if:
foo:
call bar # note: no spills for i,j,k (r1,r2,r3)
call buz
exit
And later insert spills fills on the peephole phase:
foo:
*(u64 *)(r10 - 8) = r1; # Such call pattern is
*(u64 *)(r10 - 16) = r2; # correct when used with
*(u64 *)(r10 - 24) = r3; # old kernels.
call bar
r3 = *(u64 *)(r10 - 24); # But also allows new
r2 = *(u64 *)(r10 - 16); # kernels to recognize the
r1 = *(u64 *)(r10 - 8); # pattern and remove spills/fills.
call buz
exit
The offsets for generated spills/fills are picked as minimal stack
offsets for the function. Allocated stack slots are not used for any
other purposes, in order to simplify in-kernel analysis.
In visitFREEZE we have been collecting a set/vector of
MaybePoisonOperands that later was iterated over, applying a freeze to
those operands. However, C-level fuzzy testing has discovered that the
recursiveness of ReplaceAllUsesOfValueWith may cause later operands in
the MaybePoisonOperands vector to be replaced when replacing an earlier
operand. That would then turn up as
Assertion `N1.getOpcode() != ISD::DELETED_NODE &&
"Operand is DELETED_NODE!"' failed.
failures when trying to freeze those later operands.
So we need to make sure that the vector with MaybePoisonOperands is
mutated as well when needed. Or as the solution used in this patch, make
sure to keep track of operand numbers that should be frozen instead of
having a vector of SDValues. And then we can refetch the operands while
iterating over operand numbers.
The problem was seen after adding SELECT_CC to the set of operations
including in "AllowMultipleMaybePoisonOperands". I'm not sure, but I
guess that this could happen for other operations as well for which we
allow multiple maybe poison operands.
VOPC instructions were defined with HasTrue16BitInst flag while these
true16 instructions are actually implemented with fake16 profile.
Seperate them to true16 version and fake16 version by adding
UseRealTrue16 and UseFakeTrue16 flag and fake16 instructions.
The code default to use fake16. This is preparing for the upcoming
changes in MC to support realtrue 16bit operands and vdst. The true16
and fake16 profile will be modified in the later patches.
The SPIR-V spec required basic blocks to respect some kind of ordering
(A block dominating another cannot be after in the binary layout).
---------
Signed-off-by: Nathan Gauër <brioche@google.com>
The idea behind this canonicalization is that it allows us to handle less
patterns, because we know that some will be canonicalized away. This is
indeed very useful to e.g. know that constants are always on the right.
However, this is only useful if the canonicalization is actually
reliable. This is the case for constants, but not for arguments: Moving
these to the right makes it look like the "more complex" expression is
guaranteed to be on the left, but this is not actually the case in
practice. It fails as soon as you replace the argument with another
instruction.
The end result is that it looks like things correctly work in tests,
while they actually don't. We use the "thwart complexity-based
canonicalization" trick to handle this in tests, but it's often a
challenge for new contributors to get this right, and based on the
regressions this PR originally exposed, we clearly don't get this right
in many cases.
For this reason, I think that it's better to remove this complexity
canonicalization. It will make it much easier to write tests for
commuted cases and make sure that they are handled.
This pass removes back-to-back smstart/smstop instructions
to reduce the number of streaming mode changes in a function.
The implementation as proposed doesn't aim to solve all problems
yet and suggests a number of cases that can be optimized in the
future.
This casuses assertion failures targeting 32-bit x86:
lib/Target/X86/X86RegisterInfo.cpp:989:
virtual bool llvm::X86RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator, int, unsigned int, RegScavenger *) const:
Assertion `(Is64Bit || FitsIn32Bits) && "Requesting 64-bit offset in 32-bit immediate!"' failed.
See comment on the PR.
> Fix 32-bit integer overflows in the X86 target frame layout when dealing
> with frames larger than 4gb. When this occurs, we'll scavenge a scratch
> register to be able to hold the correct stack offset for frame locals.
>
> This completes reapplying #84114.
>
> Fixes#48911
> Fixes#75944
> Fixes#87154
This reverts commit 0abb7791614947bc24931dd851ade31d02496977.
Since 2.2, `fmin.s/fmax.s` instructions follow the IEEE754-2019, if F
extension is avaiable; and `fmin.d/fmax.d` also follow the IEEE754-2019
if D extension is avaiable.
So, let's mark them as Legal.
For the tests I just added +sve instead of what actual hardware has, which is only SME,
since otherwise all the test functions need to be marked as streaming mode.
rdar://121864771
This wires up dxil-op-lower, dxil-intrinsic-expansion, dxil-translate-metadata,
and dxil-pretty-printer to the new pass manager, both as a matter of future
proofing the backend and so that they can be used more flexibly in tests.
A few arbitrary tests are updated in order to test the new PM path, and we drop
the "print-dxil-resource-md" pass since it's redundant with the pretty printer.
Pull Request: https://github.com/llvm/llvm-project/pull/104250
This allows the use a single wider operation with a restricted EVL
instead of having to split and cover via decreasing powers-of-two sizes.
On RISCV, this avoids the need for a bunch of vslidedown and vslideup
instructions to extract subvectors, and VL toggles to switch between the
various widths.
Note there is a potential downside of using vp nodes; we loose any
generic DAG combines which might have applied to the split form.
Based on experience with SelectionDAG and experimental-rv64-legal-i32, I
don't believe making s32 a legal type is viable without introducing an
invariant that s32 values are always sign extended like Mips64 does.
Mips64 does this with a separate 32-bit register class.
`experimental-rv64-legal-i32` was removed in ##102509.
This patch is part of a series to remove s32 support so we can remove
the isel patterns that SelectionDAG is no longer using. To restore code
quality, we will need to add custom W nodes like SelectionDAG.
