The peepholes in RISCVVectorPeephole need to be local and we were
failing to check if the true operand was in the same block as the
vmerge.
Fixes#110832
When adding new SEH pseudo instructions in #110024 I noticed that some
of the tests were changing their output since these new instructions
were counting towards thresholds for branching versus folding decisions.
These instructions do not result in real machine instructions being
emitted, so they should be marked as meta instructions.
SelectionDAG uses ISD::REGISTER and uses RISCV::X0 to represent VLMAX.
Then in ComplexPattern VLOpt uses selectVLOp to convert RISCV::X0 to
RISCV::VLMaxSentinel.
The original legalization patch for G_SPLAT_VECTOR used Register
RISCV::X0 directly. $x0 has no LLT type, so GIComplexOperandMatcher has
no way of matching.
The approach we are changing to here will allow us to successfully use
GIComplexOperandMatcher to implement the ComplexMatcherFn selectVLOp in
GISEL since the operand now has a type.
Certain pointer address spaces were not being correctly handled by the
GlobalISel lowering for buffer_load and buffer_store.
1. ptr addrspace(1) and addrspace(4) did not have rewrite patterns
defined for them, while p0 did, since those pointer types weren't in the
list of types that was iterated to form the patterns.
2. Vectors of pointers need to be bitcast to vectors of the
corresponding scalars, since there doesn't seem to be a good way to
define the rewrite patterns for buffer_load/store of those types
The need to bitcast vectors of pointers was also revealed to affect
ordinary `G_LOAD` and `G_STORE` in some cases, so
`shouldBitcastLoadStore()` has been fixed to handle it properly.
This is an implementation of a new "size-aware" machine block placement.
The
idea is to reorder blocks so that the number of fall-through jumps is
maximized.
Observe that profile data is ignored for the optimization, and it is
applied only
for instances with hasOptSize()=true.
This strategy has two benefits:
(i) it eliminates jump instructions, which results in smaller text size;
(ii) we avoid using profile data while reordering blocks, which yields
more
"uniform" functions, thus helping ICF and machine outliner/merger.
For large (mobile) apps, the size benefits of (i) and (ii) are roughly
the same,
combined providing up to 0.5% uncompressed and up to 1% compressed
savings size
on top of the current solution.
The optimization is turned off by default.
As of rev ea222be0d, LLVMs assembler will actually try to honour the
"fill value" part of p2align directives. X86 printed these as 0x90, which
isn't actually what it wanted: we want multi-byte nops for .text
padding. Compiling via a textual assembly file produces single-byte
nop padding since ea222be0d but the built-in assembler will produce
multi-byte nops. This divergent behaviour is undesirable.
To fix: don't set the byte padding field for x86, which allows the
assembler to pick multi-byte nops. Test that we get the same multi-byte
padding when compiled via textual assembly or directly to object file.
Added same-align-bytes-with-llasm-llobj.ll to that effect, updated
numerous other tests to not contain check-lines for the explicit padding.
RAGreedy invokes InlineSpiller to spill a particular virtreg inline.
When the spiller does this, it also identifies small, adjacent liveranges called
snippets. These are also spilled or rematerialized in the process.
However, the spiller does not inform RA that it has spilled these regs.
This means that debug variable locations referencing these regs/ranges
are lost.
Mark any spilled regs which do not have a stack slot assigned to them as
allocated to the slot being spilled to to tell LDV that those regs are
located in that slot, even though the regs might no longer exist in the
program after regalloc is finished. Also, inform RA about all of the
regs which were replaced (spilled or rematted), not just the one that was
requested so that it can properly manage the ranges of the debug vars.
This is the dual of #110144, but doesn't handle the case when the scalar
type is illegal i.e. no zfhmin/zfbfmin. It looks like softening isn't
yet implemented for insert_vector_elt operands and it will crash during
type legalization, so I've left that configuration out of the tests.
Previously we crashed because we had no lowering for f16/bf16 scalable
vectors.
Because the lowering uses vrgather_vv_vl, we need to add bf16 patterns
for it.
This patches adds a 32 bit register class for use with Zfinx instructions. This makes them more similar to F instructions and allows us to only spill 32 bits.
I've added CodeGenOnly instructions for load/store using GPRF32 as that gave better results than insert_subreg/extract_subreg.
Function arguments use this new GPRF32 register class for f32 arguments with Zfinx. Eliminating the need to use RISCVISD::FMV* nodes.
This is similar to #107446 which adds a 16 bit register class.
The motivation example likes below
```
$ cat t1.c
struct S {
int var[3];
};
int foo1 (struct S *a, struct S *b)
{
return a - b;
}
```
For cpu v1/v2/v3, the compilation will fail with the following error:
```
$ clang --target=bpf -O2 -c t1.c -mcpu=v3
t1.c:4:5: error: unsupported signed division, please convert to unsigned div/mod.
4 | int foo1 (struct S *a, struct S *b)
| ^
1 error generated.
```
The reason is that sdiv/smod is only supported at -mcpu=v4. At cpu
v1/v2/v3, only udiv/umod is supported.
But the above example (for func foo1()) is reasonable common and user
has to workaround the compilation failure by using udiv with
conditionals.
For x86, for the above t1.c, compile and dump the asm code like below:
```
$ clang -O2 -c t1.c && llvm-objdump -d t1.o
0000000000000000 <foo1>:
0: 48 29 f7 subq %rsi, %rdi
3: 48 c1 ef 02 shrq $0x2, %rdi
7: 69 c7 ab aa aa aa imull $0xaaaaaaab, %edi, %eax # imm = 0xAAAAAAAB
d: c3 retq
```
Basically sdiv can be replaced with sub, shr and imul. Latest gcc-bpf is
also able to generate code similar to x86 with -mcpu=v1. See
https://godbolt.org/z/feP9ETbjj
Generally multiplication is cheaper than divide. LLVM SelectionDAG
already supports to generate alternative codes (mul etc.) for div/mod
operations if the divisor is constant and if the cost of division is not
cheap. For BPF backend, let us use multiplication instead of divide
whenever possible. The following is the list of div/mod operations which
can be converted to mul.
- 32bit signed/unsigned div/mod with constant divisor
- 64bit signed/unsigned div with constant divisor and with 'exact' flag
in IR where 'exact' means if replacing div with mod, it is guaranteed
that modulo result will be 0. The above foo1() case belongs here.
Right now, we are printing stuff like:
zmm0 = (~zmm0 & zmm1 & mem) | (zmm0 & ~zmm1 & ~mem) |
(zmm0 & ~zmm1 & mem) | (zmm0 & zmm1 & ~mem) |
(zmm0 & zmm1 & mem)
This is not wrong but it sure is verbose and it makes it a bit hard to
understand what is going on at a glance.
The above expression is equivalent to:
zmm0 = zmm0 | (zmm1 & mem)
I considered simplifying the expressions on the fly but that gets out of
hand very quickly. Quine-McCluskey is not so simple and so fast that it
makes a ton of sense for our pretty printer. I quickly played around
with some different normal forms and ended up just using this [1] table
of functions with some minor post-processing. I made sure that each
function is identical to the appropriate BooleanFunction in Mathematica.
If the size of these tables ends up mattering for whatever reason there
are a few obvious things we could do to save space without undue effort.
[1]: https://gist.github.com/dougallj/81a80cd381988466c4e1c4889ecac95b#file-2-x86-base-txt
This reverts commit 9e45e7facbdec209a06ee20fcfbb7d0622835e21.
I found a way that these can be used. I think it's a bug somewhere
else that I need to address separately first.
PACKH is currently only selected for assembling the first two bytes of a
misligned load. A fairly complex RV32-only pattern is added for
producing PACKH+PACKH+PACK to assemble the result of a misaligned 32-bit
load.
Another pattern was added that just covers PACKH for shifted offsets 16
and 24, producing a packh and shift to replace two shifts and an 'or'.
This slightly improves RV64IZKBK for a 64-bit load, but fails to match
for the misaligned 32-bit load because the load of the upper byte is
anyext in the SelectionDAG.
I wrote the patch this way because it was quick and easy and has at
least some benefit, but the "right" approach probably merits further
discussion. Introducing target-specific SDNodes for PACK* and having
custom lowering for unaligned load/stores that introduces those nodes
them seems like it might be attractive. However, adding these patterns does provide benefit - so that's what this patch does for now.
bpf_fastcall induced spill/fill pairs should be generated for
sub-register as well as for sub-registers. At the moment this is not the
case, e.g.:
$ cat t.c
extern int foo(void) __attribute__((bpf_fastcall));
int bar(int a) {
foo();
return a;
}
$ clang --target=bpf -mcpu=v3 -O2 -S t.c -o -
...
call foo
w0 = w1
exit
Modify BPFMIPeephole.cpp:collectBPFFastCalls() to check sub-registers
liveness and thus produce correct code for example above:
*(u64 *)(r10 - 8) = r1
call foo
r1 = *(u64 *)(r10 - 8)
w0 = w1
exit
As a proxy criterion, mesa targets have unaligned-access-mode (which
determines whether the hardware allows unaligned memory accesses) not
set whereas amdhsa targets do. This PR changes tests to use amdhsa
instead of mesa and inserts additional checks with unaligned-access-mode
unset explicitly.
This is in preparation for PR #110219, which will generate different
code depending on the unaligned-access-mode.
Update VOP3 only instructions with true16 and fake16 formats.
This patch includes instructions:
V_MUL_LO_U16
V_MAX_U16
V_MAX_I16
V_MIN_U16
V_MIN_I16
V_LSHLREV_B16
V_LSHRREV_B16
V_ASHRREV_I16
Some of these check lines are insufficient to determine correctness.
Generate full check lines instead.
To reduce noise, add nounwind and use static relocation model.
As pointed out in the review of #102133, SCEVExpander currently
incorrectly reuses GEP instructions that have poison-generating flags
set. Fix this by clearing the flags on the reused instruction.
There are llvm intrinsics which we do not expect to actually represent
code after lowering or which are not implemented yet but can be found in
a customer's LLVM IR input. We do not want translation to crash when
these llvm intrinsics are found, and this PR fixes the issue with
translation crash for some known cases, aligned with Khronos Translator.
