After frontend changes in the following commit:
"BPF: preserve btf_decl_tag for parameters of extern functions"
same mechanics could be used to get the list of function parameters
and associated btf_decl_tag entries for both extern and non-extern
functions.
This commit extracts this mechanics as a separate auxiliary function
BTFDebug::processDISubprogram(). The function is called for both
extern and non-extern functions in order to generated corresponding
BTF_DECL_TAG records.
Differential Revision: https://reviews.llvm.org/D140971
Use function TargetLoweringObjectFile::SectionForGlobal() to compute
section names for globals described in BTF_KIND_DATASEC records.
This fixes a discrepancy in section name computation between
BTFDebug::processGlobals and the rest of the LLVM pipeline.
Specifically, the following example illustrates the discrepancy
before this commit:
struct Foo {
int i;
} __attribute__((aligned(16)));
struct Foo foo = { 0 };
The initializer for 'foo' looks as follows:
%struct.Foo { i32 0, [12 x i8] undef }
TargetLoweringObjectFile::SectionForGlobal() classifies 'foo' as
a part of '.bss' section, while BTFDebug::processGlobals
classified it as a part of '.data' section because of the
following expression:
SecName = Global.getInitializer()->isZeroValue() ? ".bss" : ".data"
The isZeroValue() returns false because of the undef tail of the
initializer, while SectionForGlobal() allows such patterns in '.bss'.
Differential Revision: https://reviews.llvm.org/D140505
Follow up to the series:
1. https://reviews.llvm.org/D140161
2. https://reviews.llvm.org/D140349
3. https://reviews.llvm.org/D140331
4. https://reviews.llvm.org/D140323
Completes the work from the previous two for remaining targets.
This creates the following named passes that can be run via
`llc -{start|stop}-{before|after}`:
- arc-isel
- arm-isel
- avr-isel
- bpf-isel
- csky-isel
- hexagon-isel
- lanai-isel
- loongarch-isel
- m68k-isel
- msp430-isel
- mips-isel
- nvptx-isel
- ppc-codegen
- riscv-isel
- sparc-isel
- systemz-isel
- ve-isel
- wasm-isel
- xcore-isel
A nice way to write tests for SelectionDAGISel might be to use a RUN:
line like:
llc -mtriple=<triple> -start-before=<arch>-isel -stop-after=finalize-isel -o -
Fixes: https://github.com/llvm/llvm-project/issues/59538
Reviewed By: asb, zixuan-wu
Differential Revision: https://reviews.llvm.org/D140364
This is a fairly large changeset, but it can be broken into a few
pieces:
- `llvm/Support/*TargetParser*` are all moved from the LLVM Support
component into a new LLVM Component called "TargetParser". This
potentially enables using tablegen to maintain this information, as
is shown in https://reviews.llvm.org/D137517. This cannot currently
be done, as llvm-tblgen relies on LLVM's Support component.
- This also moves two files from Support which use and depend on
information in the TargetParser:
- `llvm/Support/Host.{h,cpp}` which contains functions for inspecting
the current Host machine for info about it, primarily to support
getting the host triple, but also for `-mcpu=native` support in e.g.
Clang. This is fairly tightly intertwined with the information in
`X86TargetParser.h`, so keeping them in the same component makes
sense.
- `llvm/ADT/Triple.h` and `llvm/Support/Triple.cpp`, which contains
the target triple parser and representation. This is very intertwined
with the Arm target parser, because the arm architecture version
appears in canonical triples on arm platforms.
- I moved the relevant unittests to their own directory.
And so, we end up with a single component that has all the information
about the following, which to me seems like a unified component:
- Triples that LLVM Knows about
- Architecture names and CPUs that LLVM knows about
- CPU detection logic for LLVM
Given this, I have also moved `RISCVISAInfo.h` into this component, as
it seems to me to be part of that same set of functionality.
If you get link errors in your components after this patch, you likely
need to add TargetParser into LLVM_LINK_COMPONENTS in CMake.
Differential Revision: https://reviews.llvm.org/D137838
The value will be RAUWd, make sure the reference in CallInfo gets
updated.
It seems like this was not a problem without opaque pointers due
to the bitcast in between.
With D134950, targets get notified when a virtual register is created and/or
cloned. Targets can do the needful with the delegate callback. AMDGPU propagates
the virtual register flags maintained in the target file itself. They are useful
to identify a certain type of machine operands while inserting spill stores and
reloads. Since RegAllocFast spills the physical register itself, there is no way
its virtual register can be mapped back to retrieve the flags. It can be solved
by passing the virtual register as an additional argument. This argument has no
use when the spill interfaces are called during the greedy allocator or even the
PrologEpilogInserter and can pass a null register in such cases.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D138656
Previously we had a shared ID in SelectionDAGISel. AMDGPU has an
initializePass function for its subclass of SelectionDAGISel. No
other target does.
This causes all target specific SelectionDAGISel passes to be known
as "amdgpu-isel".
I'm not sure what would happen if another target tried to implement
an initializePass function too since the ID is already claimed.
This patch gives all targets their own ID and passes it down to
SelectionDAGISel constructor to MachineFunctionPass's constructor.
Unfortunately, I think this causes most targets to lose
print-before/after-all support for their SelectionDAGISel pass.
And they probably no longer support start/stop-before/after. We
can add initializePass functions to fix this as a follow up. NOTE:
This was probably also broken if the AMDGPU target isn't compiled in.
Step 1 to fixing PR59538.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D140161
These targets use `MCInst`, but don't explicitly link
to the library providing it (MC), and just rely on it
being pulled transitively through e.g. MCDisassembler,
but that only pulls includes, and does not link to it.
Case in point, when i add explicit destructor to `MCInst`,
defined in `.cpp`, these targets were failing to link.
Since opt no longer supports to run default (O0/O1/O2/O3/Os/Oz)
pipelines using the legacy PM, there are no in-tree uses of
TargetMachine::adjustPassManager remaining. This patch removes the
no longer used adjustPassManager functions.
Reviewed By: aeubanks
Differential Revision: https://reviews.llvm.org/D137796
Use a dyn_cast<> to IntrinsicInst and an enum compare instead.
While touching this code also re-generate the test to use positive check
lines instead of negative ones and remove some unneeded metadata.
Reviewed By: yonghong-song
Differential Revision: https://reviews.llvm.org/D138565
All in-tree targets pass pointer-sized ConstantSDNodes to the
method. This overload reduced amount of boilerplate code a bit. This
also makes getCALLSEQ_END consistent with getCALLSEQ_START, which
already takes uint64_ts.
This change completes the process of replacing OperandValueKind and OperandValueProperties which were previously passed independently in this API with a single container class which contains both.
This is the change which motivated the whole sequence which preceeded it. In an original spike version of this change, I'd noticed a nasty bug: I'd changed the signature without changing names, and as result, we silently passed additional information through a callsite which previously dropped the power-of-two fact. This might be harmless in most cases, but at least a couple clearly dependend for correctness on not passing that property through.
I did my best to split off prior changes which reduced the scope of this one, and which made it possible to use compiler assistance. For instance, every parameter which changes type in this change also changes name. This was intentional to make sure that every call site possible effected must show up in the diff. This let me audit each one closely.
D25618 added a method to verify the instruction predicates for an
emitted instruction, through verifyInstructionPredicates added into
<Target>MCCodeEmitter::encodeInstruction. This is a very useful idea,
but the implementation inside MCCodeEmitter made it only fire for object
files, not assembly which most of the llvm test suite uses.
This patch moves the code into the <Target>_MC::verifyInstructionPredicates
method, inside the InstrInfo. The allows it to be called from other
places, such as in this patch where it is called from the
<Target>AsmPrinter::emitInstruction methods which should trigger for
both assembly and object files. It can also be called from other places
such as verifyInstruction, but that is not done here (it tends to catch
errors earlier, but in reality just shows all the mir tests that have
incorrect feature predicates). The interface was also simplified
slightly, moving computeAvailableFeatures into the function so that it
does not need to be called externally.
The ARM, AMDGPU (but not R600), AVR, Mips and X86 backends all currently
show errors in the test-suite, so have been disabled with FIXME
comments.
Recommitted with some fixes for the leftover MCII variables in release
builds.
Differential Revision: https://reviews.llvm.org/D129506
When doing experiment in kernel, for kernel data structure sockptr_t
in CO-RE operation, I hit an assertion error. The sockptr_t definition
and usage look like below:
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} sockptr_t;
#pragma clang attribute pop
int test(sockptr_t *arg) {
return arg->is_kernel;
}
The assertion error looks like
clang: ../lib/Target/BPF/BPFAbstractMemberAccess.cpp:878: llvm::Value*
{anonymous}::BPFAbstractMemberAccess::computeBaseAndAccessKey(llvm::CallInst*,
{anonymous}::BPFAbstractMemberAccess::CallInfo&, std::__cxx11::string&,
llvm::MDNode*&): Assertion `TypeName.size()' failed.
In this particular, the clang frontend attach the debuginfo metadata associated
with anon structure with the preserve_access_info IR intrinsic. But the first
debuginfo type has to be a named type so libbpf can have a sound start to
do CO-RE relocation.
Besides the above approach using pragma to push attribute, the below typedef/struct
definition can have preserve_access_index directly applying to the anon struct.
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} __attribute__((preserve_access_index) sockptr_t;
This patch fixed the issue by preprocessing function argument/return types
and local variable types used by other CO-RE intrinsics. For any
typedef struct/union { ... } typedef_name
an association of <anon struct/union, typedef> is recorded to replace
the IR intrinsic metadata 'anon struct/union' to 'typedef'.
It is possible that two different 'typedef' types may have identical
anon struct/union type. For such a case, the association will be
<anon struct/union, nullptr> to indicate the invalid case.
Differential Revision: https://reviews.llvm.org/D129621
This reverts commit e2fb8c0f4b940e0285ee36c112469fa75d4b60ff as it does
not build for Release builds, and some buildbots are giving more warning
than I saw locally. Reverting to fix those issues.
D25618 added a method to verify the instruction predicates for an
emitted instruction, through verifyInstructionPredicates added into
<Target>MCCodeEmitter::encodeInstruction. This is a very useful idea,
but the implementation inside MCCodeEmitter made it only fire for object
files, not assembly which most of the llvm test suite uses.
This patch moves the code into the <Target>_MC::verifyInstructionPredicates
method, inside the InstrInfo. The allows it to be called from other
places, such as in this patch where it is called from the
<Target>AsmPrinter::emitInstruction methods which should trigger for
both assembly and object files. It can also be called from other places
such as verifyInstruction, but that is not done here (it tends to catch
errors earlier, but in reality just shows all the mir tests that have
incorrect feature predicates). The interface was also simplified
slightly, moving computeAvailableFeatures into the function so that it
does not need to be called externally.
The ARM, AMDGPU (but not R600), AVR, Mips and X86 backends all currently
show errors in the test-suite, so have been disabled with FIXME
comments.
Differential Revision: https://reviews.llvm.org/D129506
Among others, BPF currently supports the type-exists CO-RE relocation
(e.g., see D83878 & D83242). Its intention, as the name tries to convey,
is to be used for checking existence of a type in a target.
While that check is useful and has its place, we would also like to
be able to perform stricter type queries: instead of just checking mere
existence, we want to make sure that members match up in composite
types, that enum variants are present, etc. We refer to this as "type
match".
This change proposes the addition of a new relocation variant/value that
we intend to use for establishing this match relation.
Differential Revision: https://reviews.llvm.org/D126838
Current BTF only supports 32-bit value. For example,
enum T { VAL = 0xffffFFFF00000008 };
the generated BTF looks like
.long 16 # BTF_KIND_ENUM(id = 4)
.long 100663297 # 0x6000001
.long 8
.long 18
.long 8
The encoded value is 8 which equals to (uint32_t)0xffffFFFF00000008
and this is incorrect.
This patch introduced BTF_KIND_ENUM64 which permits to encode
64-bit value. The format for each enumerator looks like:
.long name_offset
.long (uint32_t)value # lower-32 bit value
.long value >> 32 # high-32 bit value
We use two 32-bit values to represent a 64-bit value as current
BTF type subsection has 4-byte alignment and gaps are not permitted
in the subsection.
This patch also added support for kflag (the bit 31 of CommonType.Info)
such that kflag = 1 implies the value is signed and kflag = 0
implies the value is unsigned. The kernel UAPI enumerator definition is
struct btf_enum {
__u32 name_off;
__s32 val;
};
so kflag = 0 with unsigned value provides backward compatability.
With this patch, for
enum T { VAL = 0xffffFFFF00000008 };
the generated BTF looks like
.long 16 # BTF_KIND_ENUM64(id = 4)
.long 3187671053 # 0x13000001
.long 8
.long 18
.long 8 # 0x8
.long 4294967295 # 0xffffffff
and the enumerator value and signedness are encoded correctly.
Differential Revision: https://reviews.llvm.org/D124641
The name `MCFixedLenDisassembler.h` is out of date after D120958.
Rename it as `MCDecoderOps.h` to reflect the change.
Reviewed By: myhsu
Differential Revision: https://reviews.llvm.org/D124987
When a specific sequence of bytes is present in the file during
disassembly the disassembler fails with the following assertion:
...
0: 18 20 00 00 00 00 00 00 lea
... Assertion `idx < size()' failed.
...
llvm::SmallVectorTemplateCommon<...>::operator[](...) ...
llvm::MCInst::getOperand(unsigned int) ...
llvm::BPFInstPrinter::printOperand(...) ...
llvm::BPFInstPrinter::printInstruction() ...
llvm::BPFInstPrinter::printInst(...) ...
...
The byte sequence causing the error is (little endian):
18 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The issue could be reproduced using the program bellow:
test.ir:
@G = constant
[16 x i8]
[i8 u0x18, i8 u0x20, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00,
i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00, i8 u0x00],
section "foo", align 8
Compiled and disassembled as follows:
cat test.ir | llc -march=bpfel -filetype=obj -o - \
| llvm-objdump --arch=bpfel --section=foo -d -
This byte sequence corresponds to FI_ri instruction declared in the
BPFInstrInfo.td as follows:
def FI_ri
: TYPE_LD_ST<BPF_IMM.Value, BPF_DW.Value,
(outs GPR:$dst),
(ins MEMri:$addr),
"lea\t$dst, $addr",
[(set i64:$dst, FIri:$addr)]> {
// This is a tentative instruction, and will be replaced
// with MOV_rr and ADD_ri in PEI phase
let Inst{51-48} = 0;
let Inst{55-52} = 2;
let Inst{47-32} = 0;
let Inst{31-0} = 0;
let BPFClass = BPF_LD;
}
Notes:
- First byte (opcode) is formed as follows:
- BPF_IMM.Value is 0x00
- BPF_DW.Value is 0x18
- BPF_LD is 0x00
- Second byte (registers) is formed as follows:
- let Inst{55-52} = 2;
- let Inst{51-48} = 0;
The FI_ri instruction is always replaced by MOV_rr ADD_ri instructions
pair in the BPFRegisterInfo::eliminateFrameIndex method. Thus, this
instruction should be invisible to disassembler. This patch achieves
this by adding "isPseudo" flag for this instruction.
The bug was found by decompiling of one of the BPF tests from Linux
kernel (llvm-objdump -D tools/testing/selftests/bpf/bpf_iter_sockmap.o)
Differential Revision: https://reviews.llvm.org/D125185
LLVM BPF pass SimplifyPatchable is used to do necessary
code conversion for CO-RE operations. When studying bpf
selftest 'exhandler', I found a corner case not handled properly.
The following is the C code, modified from original 'exhandler'
code.
int g;
int test(struct t1 *p) {
struct t2 *q = p->q;
if (q)
return 0;
struct t3 *f = q->f;
if (!f) g = 5;
return 0;
}
For code:
struct t3 *f = q->f;
if (!f) ...
The IR before BPFMISimplifyPatchable pass looks like:
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
%6:gpr = LDD killed %5:gpr, 0
%7:gpr = LDD killed %6:gpr, 0
JNE_ri killed %7:gpr, 0, %bb.3
JMP %bb.2
Note that compiler knows q = 0 based dataflow and value analysis.
The correct generated code after the pass should be
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
%7:gpr = LDD killed %5:gpr, 0
JNE_ri killed %7:gpr, 0, %bb.3
JMP %bb.2
But the current implementation did further optimization for the
above code and generates
%5:gpr = LD_imm64 @"llvm.t2:0:8$0:1"
JNE_ri killed %5:gpr, 0, %bb.3
JMP %bb.2
which is incorrect.
This patch added a cache to remember those load insns not associated
with CO-RE offset value and will skip these load insns during
transformation.
Differential Revision: https://reviews.llvm.org/D123883
Opaque pointer [1] is enabled as the default with commit [2].
Andrii found that current __builtin_preserve_enum_value() can only handle
non opaque pointer code pattern and will segfault with latest
llvm main branch where opaque-pointer is enabled by default.
This patch added the opaque pointer support.
Besides llvm selftests, also verified with bpf-next bpf selftests.
[1] https://llvm.org/docs/OpaquePointers.html
[2] https://reviews.llvm.org/D123122
Differential Revision: https://reviews.llvm.org/D123800
Delyan Kratunov reported an issue where __builtin_memcmp is
not inlined into simple load/compare instructions.
This is a known issue. In the current state, __builtin_memcmp
will be converted to memcmp call which won't work for
bpf programs.
This patch added support for expanding __builtin_memcmp with
actual loads and compares up to currently maximum 128 total loads.
The implementation is identical to PowerPC.
Differential Revision: https://reviews.llvm.org/D122676
All LLVM backends use MCDisassembler as a base class for their
instruction decoders. Use "const MCDisassembler *" for the decoder
instead of "const void *". Remove unnecessary static casts.
Reviewed By: skan
Differential Revision: https://reviews.llvm.org/D122245
The way the alignment is determined here is not compatible with
opaque pointers -- we'd have to thread it through using either
align or elementtype attributes.
However, as far as I can tell this alignment is actually never
used for this particular intrinsic, so I've dropped the assignment
entirely and converted RecordAlignment to MaybeAlign, so we get an
assertion failure if it does end up being used.
Differential Revision: https://reviews.llvm.org/D122379
When investigating an issue with bcc tool inject.py, I found
a verifier failure with latest clang. The portion of code
can be illustrated as below:
struct pid_struct {
u64 curr_call;
u64 conds_met;
u64 stack[2];
};
struct pid_struct *bpf_map_lookup_elem();
int foo() {
struct pid_struct *p = bpf_map_lookup_elem();
if (!p) return 0;
p->curr_call--;
if (p->conds_met < 1 || p->conds_met >= 3)
return 0;
if (p->stack[p->conds_met - 1] == p->curr_call)
p->conds_met--;
...
}
The verifier failure looks like:
...
8: (79) r1 = *(u64 *)(r0 +0)
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R10=fp0 fp-8=mmmm????
9: (07) r1 += -1
10: (7b) *(u64 *)(r0 +0) = r1
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
11: (79) r2 = *(u64 *)(r0 +8)
R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
12: (bf) r3 = r2
13: (07) r3 += -3
14: (b7) r4 = -2
15: (2d) if r4 > r3 goto pc+13
R0=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1=inv(id=0) R2=inv(id=2)
R3=inv(id=0,umin_value=18446744073709551614,var_off=(0xffffffff00000000; 0xffffffff))
R4=inv-2 R10=fp0 fp-8=mmmm????
16: (07) r2 += -1
17: (bf) r3 = r2
18: (67) r3 <<= 3
19: (bf) r4 = r0
20: (0f) r4 += r3
math between map_value pointer and register with unbounded min value is not allowed
Here the compiler optimized "p->conds_met < 1 || p->conds_met >= 3" to
r2 = p->conds_met
r3 = r2
r3 += -3
r4 = -2
if (r3 < r4) return 0
r2 += -1
r3 = r2
...
In the above, r3 is initially equal to r2, but is modified used by the comparison.
But later on r2 is used again. This caused verification failure.
BPF backend has a pass, AdjustOpt, to prevent such transformation, but only
focused on signed integers since typical bpf helper returns signed integers.
To fix this case, let us handle unsigned integers as well.
Differential Revision: https://reviews.llvm.org/D121937
When checking a bcc issue related to bcc tool inject.py,
I found a bug in BPFAdjustOpt pass for icmp transformation,
caused by typo's. For the following condition:
Cond2Op != ICmpInst::ICMP_SLT && Cond1Op != ICmpInst::ICMP_SLE
it should be
Cond2Op != ICmpInst::ICMP_SLT && Cond2Op != ICmpInst::ICMP_SLE
This patch fixed the problem and a test case is added.
Differential Revision: https://reviews.llvm.org/D121883
Jussi Maki reported a fatal error like below for a bitfield
CO-RE relocation:
fatal error: error in backend: Unsupported field expression for
llvm.bpf.preserve.field.info, requiring too big alignment
The failure is related to kernel struct thread_struct. The following
is a simplied example.
Suppose we have below structure:
struct t2 {
int a[8];
} __attribute__((aligned(64))) __attribute__((preserve_access_index));
struct t1 {
int f1:1;
int f2:2;
struct t2 f3;
} __attribute__((preserve_access_index));
Note that struct t2 has aligned 64, which is used sometimes in the
kernel to enforce cache line alignment.
The above struct will be encoded into BTF and the following is what
C code looks like and the struct will appear in the file like vmlinux.h.
struct t2 {
int a[8];
long: 64;
long: 64;
long: 64;
long: 64;
} __attribute__((preserve_access_index));
struct t1 {
int f1: 1;
int f2: 2;
long: 61;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
struct t2 f3;
} __attribute__((preserve_access_index));
Note that after
origin_source -> BTF -> new_source
transition, the new source has the same memory layout as the old one
but the alignment interpretation inside the compiler could be different.
The bpf program will use the later explicitly padded structure as in
vmlinux.h.
In the above case, the compiler internal ABI alignment for new struct t1
is 16 while it is 4 for old struct t1. I didn't do a thorough investigation
why the ABI alignment is 16 and I suspect it is related to anonymous padding
in the above.
Current BPF bitfield CO-RE handling requires alignment <= 8 so proper
bitfield operatin can be performed. Therefore, alignment 16 will cause
a compiler fatal error.
To fix the ABI alignment >=16, let us check whether the bitfield
can be held within a 8-byte-aligned range. If this is the case,
we can use alignment 8. Otherwise, a fatal error will be reported.
Differential Revision: https://reviews.llvm.org/D121821
This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169
