Machines with vector support handle i128 in vector registers and
therefore only have the small displacement available for memory
accesses. Update isLegalAddressingMode() to reflect this.
The PTX ISA specifies that initializers may be incomplete ([5.4.4.
Initializers](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#initializers))
> As in C, array initializers may be incomplete, i.e., the number of
initializer elements may be less than the extent of the corresponding
array dimension, with remaining array locations initialized to the
default value for the specified array type.
Emitting initializers in this form is preferable because it reduces the
size of the PTX, in some cases significantly, and can improve compile
time of ptxas as a result.
The two single source cases aren't effected by the swap or select matching
as those are dual operand specific. Similarly, a two source shuffle can't
be a rotate.
We can extend this idea for some of the shuffle types above, but some of
them are validly either single or dual source. We don't want to loose that
and the code complexity of versioning early and having to repeat some shuffle
kinds doesn't (currently) seem worth it.
Follow up to 396b6bbc, sink code into consuming branch, and fix one
comment I realized used the misleading wording. (Permute is a specific
sub-type of single source shuffle.)
This builds on bdc41106ee48dce59c500c9a3957af947f30c8c3.
This change completes the migration to a recursive shuffle lowering
strategy where when we encounter an unknown two argument shuffle, we
lower each operand as a single source permute, and then use a vselect
(i.e. a vmerge) to combine the results. This relies for code quality on
the post-isel combine which will aggressively fold that vmerge back into
the materialization of the second operand if possible.
Note: The change includes only the most immediately obvious of the
stylistic cleanup. There's a bunch of code movement that this enables
that I'll do as a separate patch as rolling it into this creates an
unreadable diff.
If we're loading a vector constant that is known to be zero in the upper elements, then attempt to shrink the constant and just scalar load the lower 32/64 bits.
Always chose the vzload/broadcast with the smallest constant load, and prefer vzload over broadcasts for same bitwidth to avoid domain flips (mainly a AVX1 issue).
Fixes#73783
The FP/SIMD instructions are optional for v8-R, so they should not be
marked as a dependency of HasV8_0rOps. This had the effect of disabling
some v8R-specific system registers when any of these features was
disabled.
I've moved these features to be enabled by default for Cortex-R82
(currently the only v8-R AArch64 core), matching the previous behavior,
and clang's default.
Based on a patch by Simi Pallipurath <simi.pallipurath@arm.com>
As otherwise SGPR+IMM instructions are not distinguishable to SGPR-only
ones in AsmParser, leading to ambiguities.
GFX12 doesn't have special SGPR-only variants, so we still allow
optional immediate offsets for the subtarget.
Also rename the offset operand classes while there.
Part of <https://github.com/llvm/llvm-project/issues/69256>.
One V_NOP or unrelated VALU instruction in between is required for
correctness when matrix A or B of current WMMA instruction overlaps with
matrix D of previous WMMA instruction.
Remaining cases of WMMA operand overlaps are handled by the hardware and
do not require handling in hazard recognizer.
Hardware may stall in cases where:
- matrix C of current WMMA instruction overlaps with matrix D of
previous WMMA instruction
- VALU instruction reads matrix D of previous WMMA instruction
- matrix A,B or C of WMMA instruction reads result of previous VALU
instruction
Implement PhiLoweringHelper for GlobalISel in DivergenceLoweringHelper.
Use machine uniformity analysis to find divergent i1 phis and select
them as lane mask phis in same way SILowerI1Copies select VReg_1 phis.
Note that divergent i1 phis include phis created by LCSSA and all cases
of uses outside of cycle are actually covered by "lowering LCSSA phis".
GlobalISel lane masks are registers with sgpr register class and S1 LLT.
TODO: General goal is that instructions created in this pass are fully
instruction-selected so that selection of lane mask phis is not split
across multiple passes.
patch 3 from: https://github.com/llvm/llvm-project/pull/73337
This patch extends HWASAN to support maintenance of debug-info that
isn't stored as intrinsics, but is instead in a DPValue object. This is
straight-forwards: we collect any such objects in StackInfoBuilder, and
apply the same operations to them as we would to dbg.value and similar
intrinsics.
I've also replaced some calls to getNextNode with debug-info skipping
next calls, and use iterators for instruction insertion rather than
instruction pointers. This avoids any difference in output between
intrinsic / non-intrinsic debug-info, but also means that any debug-info
comes before code inserted by HWAsan, rather than afterwards. See the
test modifications, where the variable assignment (presented as a
dbg.value) jumps up over all the code inserted by HWAsan. Seeing how the
code inserted by HWAsan is always (AFAIUI) given the source-location of
the instruction being instrumented, I don't believe this will have any
effect on which lines variable assignments become visible on; it may
extend the number of instructions covered by the assignments though.
ADC/SBB with reverse encoding is never emitted by compiler before
encoding optimization, which is called after flag-copy lowering.
This is a partial reland for 8bbf100799a97f8342bf1a8409c6fb48f03e837f
The current comment was written whenever we had separate TU/TA variants
for
each pseudo, and hasn't been accurate for a while.
This method has grown rather complicated over time so rather than
enumerate all
the different possible cases now (which must be a lot), this updates the
comment to list the different rules that are required for us to be able
to fold
a vmerge.
This patch allows VCIX instructions that have side effect to be
reordered
with memory and other side effecting instructions. However we don't want
VCIX instructions to be reordered with each other, so we propose a dummy
register called VCIX_STATE and make these instructions implicitly define
and use
it.
`CodeGenPassBuilder` is not very tightly coupled to CodeGen, it may need
to reference some method in pass builder in future, so move
`CodeGenPassBuilder.h` to Passes.
CSR SGPR spilling currently uses the early available physical VGPRs. It
currently imposes a high register pressure while trying to allocate
large VGPR tuples within the default register budget.
This patch changes the spilling strategy by picking the VGPRs in the
reverse order, the highest available VGPR first and later after regalloc
shift them back to the lowest available range. With that, the initial
VGPRs would be available for allocation and possibility
of finding large number of contiguous registers will be more.
Add new pass manager support to `llc`. Users can use
`--passes=pass1,pass2...` to run mir passes, and use `--enable-new-pm`
to run default codegen pipeline.
This patch is taken from [D83612](https://reviews.llvm.org/D83612), the
original author is @yuanfang-chen.
---------
Co-authored-by: Yuanfang Chen <455423+yuanfang-chen@users.noreply.github.com>
Refer to RISCV, we will fix up the alignment if linker relaxation
changes code size and breaks alignment. Insert enough Nops and emit
R_LARCH_ALIGN relocation type so that linker could satisfy the alignment
by removing Nops.
It does so only in sections with the SHF_EXECINSTR flag.
In LoongArch psABI v2.30, R_LARCH_ALIGN requires symbol index. The
lowest 8 bits of addend represent alignment and the other bits of addend
represent the maximum number of bytes to emit.
This patch adds basic TLSDESC support in the RISC-V backend.
Specifically, we add new relocation types for TLSDESC, as prescribed in
https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/373, and add a
new pseudo instruction to simplify code generation.
This patch does not try to optimize the local dynamic case, which can be
improved in separate patches.
Linker side changes will also be handled separately.
The current implementation is only enabled when passing the new
`-enable-tlsdesc` codegen flag.
There are matching store opcodes (stfd, stxsiwx) for the load opcodes
that make 32-bit and 64-bit vector operations cheap with VSX, so stores
should also be cheap.
This is the first step towards an alternate shuffle lowering design for
the general two vector argument case. The goal is to leverage the
existing lowering for single vector permutes to avoid as many of the
vrgathers as required - even if we do need the other.
This patch handles only the first argument, and is arguably a slightly
weird half-step. However, the test changes from the full two argument
recurse patch are a lot harder to reason about. Taking this half step
gives much more easily reviewable changes, and is thus worthwhile. I
intend to post the patch for the second argument once this has landed.
If we have a shuffle which is larger than m1, we may be able to split it
into a series of individual m1 shuffles. This patch starts with the
subcase where the mask allows a 1-to-1 mapping from source register to
destination register - each with a possible permutation of their own. We
can potentially extend this later, thought in practice this seems to
already catch a number of the most interesting cases.
int_amdgcn_global_load_tr did not specify non-temporal load transpose,
thus we should
not genetrate the non-temporal hint for the load. We need to implement
getTgtMemIntrinsic
to create the corresponding MemSDNode. And we don't set the non-temporal
flag because
the intrinsic did not specify it.
NOTE: We need to implement getTgtMemIntrinsic for any memory intrinsics.
AssemblerPredicate was almost always indented to the same column as the
first operand to Predicate. But AssemblerPredicate is a separate base
class so should have the same indentation as Predicate.
For the string passed to AssemblePredicate, I aligned it to the other
arguments on the previous if it fit in 80 columns. Otherwise I indented
4 spaces past the start of AssemblerPredicate.
For some vendor extensions I put the 2 classes on new lines instead of
the same line as the def. This gave more room for the strings and was
more consistent with other formatting in that portion of the file.
Minor rework of the fallback case for two argument shuffles in lowerVECTOR_SHUFFLE. We had some common code which wasn't actually common, and simplified significantly once specialized for whether we had a select or not.
Make Candidate's front() and back() functions return references to
MachineInstr and introduce begin() and end() returning iterators, the
same way it is usually done in other container-like classes.
This makes possible to iterate over the instructions contained in
Candidate the same way one can iterate over MachineBasicBlock (note that
begin() and end() return bundled iterators, just like MachineBasicBlock
does, but no instr_begin() and instr_end() are defined yet).
