468 Commits

Author SHA1 Message Date
Yonghong Song
5898979387 BPF: support inlining __builtin_memcmp intrinsic call
Delyan Kratunov reported an issue where __builtin_memcmp is
not inlined into simple load/compare instructions.
This is a known issue. In the current state, __builtin_memcmp
will be converted to memcmp call which won't work for
bpf programs.

This patch added support for expanding __builtin_memcmp with
actual loads and compares up to currently maximum 128 total loads.
The implementation is identical to PowerPC.

Differential Revision: https://reviews.llvm.org/D122676
2022-03-29 15:03:26 -07:00
Maksim Panchenko
4ae9745af1 [Disassember][NFCI] Use strong type for instruction decoder
All LLVM backends use MCDisassembler as a base class for their
instruction decoders. Use "const MCDisassembler *" for the decoder
instead of "const void *". Remove unnecessary static casts.

Reviewed By: skan

Differential Revision: https://reviews.llvm.org/D122245
2022-03-25 18:53:59 -07:00
Nikita Popov
c48b4641c7 [BPF] Don't fetch alignment of llvm.preserve.union.access.index argument
The way the alignment is determined here is not compatible with
opaque pointers -- we'd have to thread it through using either
align or elementtype attributes.

However, as far as I can tell this alignment is actually never
used for this particular intrinsic, so I've dropped the assignment
entirely and converted RecordAlignment to MaybeAlign, so we get an
assertion failure if it does end up being used.

Differential Revision: https://reviews.llvm.org/D122379
2022-03-25 09:53:30 +01:00
Yonghong Song
2e94d8e67a [BPF] handle unsigned icmp ops in BPFAdjustOpt pass
When investigating an issue with bcc tool inject.py, I found
a verifier failure with latest clang. The portion of code
can be illustrated as below:
  struct pid_struct {
    u64 curr_call;
    u64 conds_met;
    u64 stack[2];
  };
  struct pid_struct *bpf_map_lookup_elem();
  int foo() {
    struct pid_struct *p = bpf_map_lookup_elem();
    if (!p) return 0;
    p->curr_call--;
    if (p->conds_met < 1 || p->conds_met >= 3)
        return 0;
    if (p->stack[p->conds_met - 1] == p->curr_call)
        p->conds_met--;
    ...
  }

The verifier failure looks like:
  ...
  8: (79) r1 = *(u64 *)(r0 +0)
   R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R10=fp0 fp-8=mmmm????
  9: (07) r1 += -1
  10: (7b) *(u64 *)(r0 +0) = r1
   R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
  11: (79) r2 = *(u64 *)(r0 +8)
   R0_w=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1_w=inv(id=0) R10=fp0 fp-8=mmmm????
  12: (bf) r3 = r2
  13: (07) r3 += -3
  14: (b7) r4 = -2
  15: (2d) if r4 > r3 goto pc+13
   R0=map_value(id=0,off=0,ks=4,vs=32,imm=0) R1=inv(id=0) R2=inv(id=2)
   R3=inv(id=0,umin_value=18446744073709551614,var_off=(0xffffffff00000000; 0xffffffff))
   R4=inv-2 R10=fp0 fp-8=mmmm????
  16: (07) r2 += -1
  17: (bf) r3 = r2
  18: (67) r3 <<= 3
  19: (bf) r4 = r0
  20: (0f) r4 += r3
  math between map_value pointer and register with unbounded min value is not allowed

Here the compiler optimized "p->conds_met < 1 || p->conds_met >= 3" to
  r2 = p->conds_met
  r3 = r2
  r3 += -3
  r4 = -2
  if (r3 < r4) return 0
  r2 += -1
  r3 = r2
  ...
In the above, r3 is initially equal to r2, but is modified used by the comparison.
But later on r2 is used again. This caused verification failure.

BPF backend has a pass, AdjustOpt, to prevent such transformation, but only
focused on signed integers since typical bpf helper returns signed integers.
To fix this case, let us handle unsigned integers as well.

Differential Revision: https://reviews.llvm.org/D121937
2022-03-17 16:24:39 -07:00
Yonghong Song
d2b4a675a8 [BPF] Fix a bug in BPFAdjustOpt pass for icmp transformation
When checking a bcc issue related to bcc tool inject.py,
I found a bug in BPFAdjustOpt pass for icmp transformation,
caused by typo's. For the following condition:
  Cond2Op != ICmpInst::ICMP_SLT && Cond1Op != ICmpInst::ICMP_SLE
it should be
  Cond2Op != ICmpInst::ICMP_SLT && Cond2Op != ICmpInst::ICMP_SLE

This patch fixed the problem and a test case is added.

Differential Revision: https://reviews.llvm.org/D121883
2022-03-17 09:25:18 -07:00
Yonghong Song
98e2274458 [BPF] fix a CO-RE bitfield relocation error with >8 record alignment
Jussi Maki reported a fatal error like below for a bitfield
CO-RE relocation:
  fatal error: error in backend: Unsupported field expression for
  llvm.bpf.preserve.field.info, requiring too big alignment
The failure is related to kernel struct thread_struct. The following
is a simplied example.

Suppose we have below structure:
  struct t2 {
    int a[8];
  } __attribute__((aligned(64))) __attribute__((preserve_access_index));
  struct t1 {
    int f1:1;
    int f2:2;
    struct t2 f3;
  } __attribute__((preserve_access_index));

Note that struct t2 has aligned 64, which is used sometimes in the
kernel to enforce cache line alignment.

The above struct will be encoded into BTF and the following is what
C code looks like and the struct will appear in the file like vmlinux.h.
  struct t2 {
        int a[8];
        long: 64;
        long: 64;
        long: 64;
        long: 64;
  } __attribute__((preserve_access_index));
  struct t1 {
        int f1: 1;
        int f2: 2;
        long: 61;
        long: 64;
        long: 64;
        long: 64;
        long: 64;
        long: 64;
        long: 64;
        long: 64;
        struct t2 f3;
  } __attribute__((preserve_access_index));

Note that after
  origin_source -> BTF -> new_source
transition, the new source has the same memory layout as the old one
but the alignment interpretation inside the compiler could be different.
The bpf program will use the later explicitly padded structure as in
vmlinux.h.

In the above case, the compiler internal ABI alignment for new struct t1
is 16 while it is 4 for old struct t1. I didn't do a thorough investigation
why the ABI alignment is 16 and I suspect it is related to anonymous padding
in the above.

Current BPF bitfield CO-RE handling requires alignment <= 8 so proper
bitfield operatin can be performed. Therefore, alignment 16 will cause
a compiler fatal error.

To fix the ABI alignment >=16, let us check whether the bitfield
can be held within a 8-byte-aligned range. If this is the case,
we can use alignment 8. Otherwise, a fatal error will be reported.

Differential Revision: https://reviews.llvm.org/D121821
2022-03-16 12:16:46 -07:00
serge-sans-paille
989f1c72e0 Cleanup codegen includes
This is a (fixed) recommit of https://reviews.llvm.org/D121169

after:  1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681
2022-03-16 08:43:00 +01:00
serge-sans-paille
ed98c1b376 Cleanup includes: DebugInfo & CodeGen
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332
2022-03-12 17:26:40 +01:00
Nico Weber
a278250b0f Revert "Cleanup codegen includes"
This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169
2022-03-10 07:59:22 -05:00
serge-sans-paille
7f230feeea Cleanup codegen includes
after:  1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169
2022-03-10 10:00:30 +01:00
serge-sans-paille
3c4410dfca Cleanup includes: LLVMTarget
Most notably, Pass.h is no longer included by TargetMachine.h
before: 1063570306
after:  1063332844

Differential Revision: https://reviews.llvm.org/D121168
2022-03-10 10:00:29 +01:00
Jameson Nash
c4b1a63a1b mark getTargetTransformInfo and getTargetIRAnalysis as const
Seems like this can be const, since Passes shouldn't modify it.

Reviewed By: wsmoses

Differential Revision: https://reviews.llvm.org/D120518
2022-02-25 14:30:44 -05:00
Nikita Popov
87ebd9a36f [IR] Use CallBase::getParamElementType() (NFC)
As this method now exists on CallBase, use it rather than the
one on AttributeList.
2022-02-25 10:01:58 +01:00
Yonghong Song
3671bdbcd2 [BPF] Fix a BTF type pruning bug
In BPF backend, BTF type generation may skip
some debuginfo types if they are the pointee
type of a struct member. For example,
  struct task_struct {
    ...
    struct mm_struct                *mm;
    ...
  };
BPF backend may generate a forward decl for
'struct mm_struct' instead of full type if
there are no other usage of 'struct mm_struct'.
The reason is to avoid bringing too much unneeded types
in BTF.

Alexei found a pruning bug where we may miss
some full type generation. The following is an illustrating
example:
   struct t1 { ... }
   struct t2 { struct t1 *p; };
   struct t2 g;
   void foo(struct t1 *arg) { ... }
In the above case, we will have partial debuginfo chain like below:
   struct t2 -> member p
                        \ -> ptr -> struct t1
                        /
     foo -> argument arg
During traversing
   struct t2 -> member p -> ptr -> struct t1
The corresponding BTF types are generated except 'struct t1' which
will be in FixUp stage. Later, when traversing
   foo -> argument arg -> ptr -> struct t1
The 'ptr' BTF type has been generated and currently implementation
ignores 'pointer' type hence 'struct t1' is not generated.

This patch fixed the issue not just for the above case, but for
general case with multiple derived types, e.g.,
   struct t2 -> member p
                        \ -> const -> ptr -> volatile -> struct t1
                        /
     foo -> argument arg

Differential Revision: https://reviews.llvm.org/D119986
2022-02-16 17:23:34 -08:00
Shao-Ce SUN
2aed07e96c [NFC][MC] remove unused argument MCRegisterInfo in MCCodeEmitter
Reviewed By: skan

Differential Revision: https://reviews.llvm.org/D119846
2022-02-16 13:10:09 +08:00
Yonghong Song
f419029fcd [BPF] Fix a bug in BTF_KIND_TYPE_TAG generation
Kumar Kartikeya Dwivedi reported a bug ([1]) where BTF_KIND_TYPE_TAG types
are not generated.

Currently, BPF backend only generates BTF types which are used by
the program, e.g., global variables, functions and some builtin functions.
For example, suppose we have
  struct task_struct {
    ...
    struct task_group               *sched_task_group;
    struct mm_struct                *mm;
    ...
    pid_t                           pid;
    pid_t                           tgid;
    ...
  }
If BPF program intends to access task_struct->pid and task_struct->tgid,
there really no need to generate BTF types for struct task_group
and mm_struct.

In BPF backend, during BTF generation, when generating BTF for struct
task_struct, if types for task_group and mm_struct have not been generated
yet, a Fixup structure will be created, which will be reexamined later
to instantiate into either a full type or a forward type.

In current implementation, if we have something like
  struct foo {
     struct bar  __tag1    *f;
  };
and when generating types for struct foo, struct bar type
has not been generated, the __tag1 will be lost during later
Fixup instantiation. This patch fixed this issue by properly
handling btf_type_tag's during Fixup instantiation stage.

  [1] https://lore.kernel.org/bpf/20220210232411.pmhzj7v5uptqby7r@apollo.legion/

Differential Revision: https://reviews.llvm.org/D119799
2022-02-14 19:43:57 -08:00
serge-sans-paille
06943537d9 Cleanup MCParser headers
As usual with that header cleanup series, some implicit dependencies now need to
be explicit:

llvm/MC/MCParser/MCAsmParser.h no longer includes llvm/MC/MCParser/MCAsmLexer.h

Preprocessed lines to build llvm on my setup:
after:  1068185081
before: 1068324320

So no compile time benefit to expect, but we still get the looser coupling
between files which is great.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119359
2022-02-11 10:39:29 +01:00
serge-sans-paille
ef736a1c39 Cleanup LLVMMC headers
There's a few relevant forward declarations in there that may require downstream
adding explicit includes:

llvm/MC/MCContext.h no longer includes llvm/BinaryFormat/ELF.h, llvm/MC/MCSubtargetInfo.h, llvm/MC/MCTargetOptions.h
llvm/MC/MCObjectStreamer.h no longer include llvm/MC/MCAssembler.h
llvm/MC/MCAssembler.h no longer includes llvm/MC/MCFixup.h, llvm/MC/MCFragment.h

Counting preprocessed lines required to rebuild llvm-project on my setup:
before: 1052436830
after:  1049293745

Which is significant and backs up the change in addition to the usual benefits of
decreasing coupling between headers and compilation units.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119244
2022-02-09 11:09:17 +01:00
serge-sans-paille
ffe8720aa0 Reduce dependencies on llvm/BinaryFormat/Dwarf.h
This header is very large (3M Lines once expended) and was included in location
where dwarf-specific information were not needed.

More specifically, this commit suppresses the dependencies on
llvm/BinaryFormat/Dwarf.h in two headers: llvm/IR/IRBuilder.h and
llvm/IR/DebugInfoMetadata.h. As these headers (esp. the former) are widely used,
this has a decent impact on number of preprocessed lines generated during
compilation of LLVM, as showcased below.

This is achieved by moving some definitions back to the .cpp file, no
performance impact implied[0].

As a consequence of that patch, downstream user may need to manually some extra
files:

llvm/IR/IRBuilder.h no longer includes llvm/BinaryFormat/Dwarf.h
llvm/IR/DebugInfoMetadata.h no longer includes llvm/BinaryFormat/Dwarf.h

In some situations, codes maybe relying on the fact that
llvm/BinaryFormat/Dwarf.h was including llvm/ADT/Triple.h, this hidden
dependency now needs to be explicit.

$ clang++ -E  -Iinclude -I../llvm/include ../llvm/lib/Transforms/Scalar/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l
after:   10978519
before:  11245451

Related Discourse thread: https://llvm.discourse.group/t/include-what-you-use-include-cleanup
[0] https://llvm-compile-time-tracker.com/compare.php?from=fa7145dfbf94cb93b1c3e610582c495cb806569b&to=995d3e326ee1d9489145e20762c65465a9caeab4&stat=instructions

Differential Revision: https://reviews.llvm.org/D118781
2022-02-04 11:44:03 +01:00
Nikita Popov
fc72f3a168 [BTFDebug] Avoid pointer element type access
Use the global value type instead.
2022-01-27 10:30:21 +01:00
Nikita Popov
aa97bc116d [NFC] Remove uses of PointerType::getElementType()
Instead use either Type::getPointerElementType() or
Type::getNonOpaquePointerElementType().

This is part of D117885, in preparation for deprecating the API.
2022-01-25 09:44:52 +01:00
Kazu Hirata
f3a344d212 [Target] Remove redundant member initialization (NFC)
Identified with readability-redundant-member-init.
2022-01-06 22:01:44 -08:00
Kazu Hirata
e5947760c2 Revert "[llvm] Remove redundant member initialization (NFC)"
This reverts commit fd4808887ee47f3ec8a030e9211169ef4fb094c3.

This patch causes gcc to issue a lot of warnings like:

  warning: base class ‘class llvm::MCParsedAsmOperand’ should be
  explicitly initialized in the copy constructor [-Wextra]
2022-01-03 11:28:47 -08:00
Kazu Hirata
7e163afd9e Remove redundant void arguments (NFC)
Identified by modernize-redundant-void-arg.
2022-01-02 10:20:19 -08:00
Kazu Hirata
fd4808887e [llvm] Remove redundant member initialization (NFC)
Identified with readability-redundant-member-init.
2022-01-01 16:18:18 -08:00
Kazu Hirata
5a667c0e74 [llvm] Use nullptr instead of 0 (NFC)
Identified with modernize-use-nullptr.
2021-12-28 08:52:25 -08:00
Yonghong Song
76b7d73429 BPF: report better error message for BTF_TYPE_ID_REMOTE relo failure
Matteo Croce reported a bpf backend fatal error in
https://github.com/llvm/llvm-project/issues/52779

A simplified case looks like:
  $ cat bug.c
  extern int do_smth(int);
  int test() {
    return __builtin_btf_type_id(*(typeof(do_smth) *)do_smth, 1);
  }
  $ clang -target bpf -O2 -g -c bug.c
  fatal error: error in backend: Empty type name for BTF_TYPE_ID_REMOTE reloc
  ...

The reason for the fatal error is that the relocation is against
a DISubroutineType like type 13 below:
  !10 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
  !11 = !{}
  !12 = !DILocation(line: 3, column: 10, scope: !7)
  !13 = !DISubroutineType(types: !14)
  !14 = !{!10, !10}

The DISubroutineType doesn't have a name and there is no way for
downstream bpfloader/kernel to do proper relocation for it.

But we can improve error message to be more specific for this case.
The patch improved the error message to be:
  fatal error: error in backend: SubroutineType not supported for BTF_TYPE_ID_REMOTE reloc

Differential Revision: https://reviews.llvm.org/D116063
2021-12-20 21:06:19 -08:00
Kazu Hirata
562356d6e3 [Target] Use range-based for loops (NFC) 2021-11-26 08:23:01 -08:00
Yonghong Song
8fb3f84484 BPF: Workaround InstCombine trunc+icmp => mask+icmp Optimization
Patch [1] added further InstCombine trunc+icmp => mask+icmp
optimization and this caused a couple of bpf selftest failure.
Previous llvm BPF backend patch [2] introduced llvm.bpf.compare
builtin to handle such situations.

This patch further added support ">" and ">=" icmp opcodes.
Tested with bpf selftests and all tests are passed including two
previously failed ones.

Note Patch [1] also added optimization if the to-be-compared
constant is negative-power-of-2 (-C) or not-of-power-of-2 (~C).
This patch didn't implement these two cases as typical bpf
program compares a scalar to a positive length or boundary value,
and this scalar later is used as a index into an array buffer
or packet buffer.

  [1] https://reviews.llvm.org/D112634
  [2] https://reviews.llvm.org/D112938

Differential Revision: https://reviews.llvm.org/D114215
2021-11-18 20:25:28 -08:00
Zarko Todorovski
5b8bbbecfa [NFC][llvm] Inclusive language: reword and remove uses of sanity in llvm/lib/Target
Reworded removed code comments that contain `sanity check` and `sanity
test`.
2021-11-17 21:59:00 -05:00
Yonghong Song
8d499bd5bc BPF: change btf_type_tag BTF output format
For the declaration like below:
  int __tag1 * __tag1 __tag2 *g
Commit 41860e602aaa ("BPF: Support btf_type_tag attribute")
implemented the following encoding:
  VAR(g) -> __tag1 --> __tag2 -> pointer -> __tag1 -> pointer -> int

Some further experiments with linux btf_type_tag support, esp.
with generating attributes in vmlinux.h, and also some internal
discussion showed the following format is more desirable:
  VAR(g) -> pointer -> __tag2 -> __tag1 -> pointer -> __tag1 -> int

The format makes it similar to other modifier like 'const', e.g.,
  const int *g
which has encoding VAR(g) -> PTR -> CONST -> int

Differential Revision: https://reviews.llvm.org/D113496
2021-11-09 11:34:25 -08:00
Kazu Hirata
2c4ba3e9d3 [Target] Use make_early_inc_range (NFC) 2021-11-05 09:14:32 -07:00
Yonghong Song
41860e602a BPF: Support btf_type_tag attribute
A new kind BTF_KIND_TYPE_TAG is defined. The tags associated
with a pointer type are emitted in their IR order as modifiers.
For example, for the following declaration:
  int __tag1 * __tag1 __tag2 *g;
The BTF type chain will look like
  VAR(g) -> __tag1 --> __tag2 -> pointer -> __tag1 -> pointer -> int
In the above "->" means BTF CommonType.Type which indicates
the point-to type.

Differential Revision: https://reviews.llvm.org/D113222
2021-11-04 17:01:36 -07:00
Yonghong Song
f63405f6e3 BPF: Workaround an InstCombine ICmp transformation with llvm.bpf.compare builtin
Commit acabad9ff6bf ("[InstCombine] try to canonicalize icmp with
trunc op into mask and cmp") added a transformation to
convert "(conv)a < power_2_const" to "a & <const>" in certain
cases and bpf kernel verifier has to handle the resulted code
conservatively and this may reject otherwise legitimate program.

This commit tries to prevent such a transformation. A bpf backend
builtin llvm.bpf.compare is added. The ICMP insn, which is subject to
above InstCombine transformation, is converted to the builtin
function. The builtin function is later lowered to original ICMP insn,
certainly after InstCombine pass.

With this change, all affected bpf strobemeta* selftests are
passed now.

Differential Revision: https://reviews.llvm.org/D112938
2021-11-01 14:46:20 -07:00
Yonghong Song
0472e83ffc BPF: emit BTF_KIND_DECL_TAG for typedef types
If a typedef type has __attribute__((btf_decl_tag("str"))) with
bpf target, emit BTF_KIND_DECL_TAG for that type in the BTF.

Differential Revision: https://reviews.llvm.org/D112259
2021-10-21 12:09:42 -07:00
Yonghong Song
cd40b5a712 BPF: set .BTF and .BTF.ext section alignment to 4
Currently, .BTF and .BTF.ext has default alignment of 1.
For example,
  $ cat t.c
    int foo() { return 0; }
  $ clang -target bpf -O2 -c -g t.c
  $ llvm-readelf -S t.o
    ...
    Section Headers:
    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    ...
    [ 7] .BTF              PROGBITS        0000000000000000 000167 00008b 00      0   0  1
    [ 8] .BTF.ext          PROGBITS        0000000000000000 0001f2 000050 00      0   0  1

But to have no misaligned data access, .BTF and .BTF.ext
actually requires alignment of 4. Misalignment is not an issue
for architecture like x64/arm64 as it can handle it well. But
some architectures like mips may incur a trap if .BTF/.BTF.ext
is not properly aligned.

This patch explicitly forced .BTF and .BTF.ext alignment to be 4.
For the above example, we will have
    [ 7] .BTF              PROGBITS        0000000000000000 000168 00008b 00      0   0  4
    [ 8] .BTF.ext          PROGBITS        0000000000000000 0001f4 000050 00      0   0  4

Differential Revision: https://reviews.llvm.org/D112106
2021-10-19 16:26:01 -07:00
Yonghong Song
f4a8526cc4 [NFC][BPF] fix comments and rename functions related to BTF_KIND_DECL_TAG
There are no functionality change.
Fix some comments and rename processAnnotations() to
processDeclAnnotations() to avoid confusion when later
BTF_KIND_TYPE_TAG is introduced (https://reviews.llvm.org/D111199).
2021-10-18 10:43:45 -07:00
Yonghong Song
e9e4fc0fd3 BPF: fix a bug in IRPeephole pass
Commit 009f3a89d833 ("BPF: remove intrindics @llvm.stacksave()
and @llvm.stackrestore()") implemented IRPeephole pass to remove
llvm.stacksave()/stackrestore() instrinsics.
Buildbot reported a failure:
  UNREACHABLE executed at ../lib/IR/LegacyPassManager.cpp:1445!
which is:
  llvm_unreachable("Pass modifies its input and doesn't report it");

The code has changed but the implementation didn't return true
for changing. This patch fixed this problem.
2021-10-18 10:18:24 -07:00
Yonghong Song
009f3a89d8 BPF: remove intrindics @llvm.stacksave() and @llvm.stackrestore()
Paul Chaignon reported a bpf verifier failure ([1]) due to using
non-ABI register R11. For the test case, llvm11 is okay while
llvm12 and later generates verifier unfriendly code.

The failure is related to variable length array size.
The following mimics the variable length array definition
in the test case:

struct t { char a[20]; };
void foo(void *);
int test() {
   const int a = 8;
   char tmp[AA + sizeof(struct t) + a];
   foo(tmp);
   ...
}

Paul helped bisect that the following llvm commit is
responsible:

552c6c232872 ("PR44406: Follow behavior of array bound constant
              folding in more recent versions of GCC.")

Basically, before the above commit, clang frontend did constant
folding for array size "AA + sizeof(struct t) + a" to be 68,
so used alloca for stack allocation. After the above commit,
clang frontend didn't do constant folding for array size
any more, which results in a VLA and llvm.stacksave/llvm.stackrestore
is generated.

BPF architecture API does not support stack pointer (sp) register.
The LLVM internally used R11 to indicate sp register but it should
not be in the final code. Otherwise, kernel verifier will reject it.

The early patch ([2]) tried to fix the issue in clang frontend.
But the upstream discussion considered frontend fix is really a
hack and the backend should properly undo llvm.stacksave/llvm.stackrestore.
This patch implemented a bpf IR phase to remove these intrinsics
unconditionally. If eventually the alloca can be resolved with
constant size, r11 will not be generated. If alloca cannot be
resolved with constant size, SelectionDag will complain, the same
as without this patch.

 [1] https://lore.kernel.org/bpf/20210809151202.GB1012999@Mem/
 [2] https://reviews.llvm.org/D107882

Differential Revision: https://reviews.llvm.org/D111897
2021-10-18 09:51:19 -07:00
Yonghong Song
1321e47298 BPF: rename BTF_KIND_TAG to BTF_KIND_DECL_TAG
Per discussion in https://reviews.llvm.org/D111199,
the existing btf_tag attribute will be renamed to
btf_decl_tag. This patch updated BTF backend to
use btf_decl_tag attribute name and also
renamed BTF_KIND_TAG to BTF_KIND_DECL_TAG.

Differential Revision: https://reviews.llvm.org/D111592
2021-10-11 21:33:39 -07:00
Reid Kleckner
b3a6d096d7 Fix shlib builds for all lib/Target/*/TargetInfo libs
They all must depend on MC now that the target registry is in MC.
Also fix llvm-cxxdump
2021-10-08 15:21:13 -07:00
Reid Kleckner
89b57061f7 Move TargetRegistry.(h|cpp) from Support to MC
This moves the registry higher in the LLVM library dependency stack.
Every client of the target registry needs to link against MC anyway to
actually use the target, so we might as well move this out of Support.

This allows us to ensure that Support doesn't have includes from MC/*.

Differential Revision: https://reviews.llvm.org/D111454
2021-10-08 14:51:48 -07:00
Simon Pilgrim
21661607ca [llvm] Replace report_fatal_error(std::string) uses with report_fatal_error(Twine)
As described on D111049, we're trying to remove the <string> dependency from error handling and replace uses of report_fatal_error(const std::string&) with the Twine() variant which can be forward declared.
2021-10-06 12:04:30 +01:00
Yonghong Song
3562ad3ebe BPF: implement isLegalAddressingMode() properly
Latest upstream llvm caused the kernel bpf selftest emitting the
following warnings:

  In file included from progs/profiler3.c:6:
  progs/profiler.inc.h:489:2: warning: loop not unrolled:
    the optimizer was unable to perform the requested transformation;
    the transformation might be disabled or specified as part of an unsupported
    transformation ordering [-Wpass-failed=transform-warning]
          for (int i = 0; i < MAX_PATH_DEPTH; i++) {
          ^

Further bisecting shows this SimplifyCFG patch ([1]) changed
the condition on how to fold branch to common dest. This caused
some unroll pragma is not honored in selftests/bpf.

The patch [1] test getUserCost() as the condition to
perform the certain basic block folding transformation.
For the above example, before the loop unroll pass, the control flow
looks like:
    cond_block:
       branch target: body_block, cleanup_block
    body_block:
       branch target: cleanup_block, end_block
    end_block:
       branch target: cleanup_block, end10_block
    end10_block:
       %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2
       %inc = add nuw nsw i32 %i.0, 1
       branch target: cond_block

In the above, %call2 is an unknown scalar.

Before patch [1], end10_block will be folded into end_block, forming
the code like
    cond_block:
       branch target: body_block, cleanup_block
    body_block:
       branch target: cleanup_block, end_block
    end_block:
       branch target: cleanup_block, cond_block
and the compiler is happy to perform unrolling.

With patch [1], getUserCost(), which calls getGEPCost(), which calls
isLegalAddressingMode() in TargetLoweringBase.cpp, considers IR
  %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2
is free, so the above basic block folding transformation is not performed
and unrolling does not happen.

For BPF target, the IR
  %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2
is not free and we don't have ld/st instruction address with 'r+r' mode.

This patch implemented a BPF hook for isLegalAddressingMode(), which is
identical to Mips isLegalAddressingMode() implementation where
the address pattern like 'r+r', 'r+r+i' or '2*r' are not allowed.
With testing kernel bpf selftests, all loop not unrolled warnings
are gone and all selftests run successfully.

  [1] https://reviews.llvm.org/D108837

Differential Revision: https://reviews.llvm.org/D110789
2021-09-30 16:41:15 -07:00
Simon Pilgrim
b1f38a27f0 [Target][CodeGen] Remove default CostKind arguments on inner/impl TTI overrides
Based off a discussion on D110100, we should be avoiding default CostKinds whenever possible.

This initial patch removes them from the 'inner' target implementation callbacks - these should only be used by the main TTI calls, so this should guarantee that we don't cause changes in CostKind by missing it in an inner call. This exposed a few missing arguments in getGEPCost and reduction cost calls that I've cleaned up.

Differential Revision: https://reviews.llvm.org/D110242
2021-09-22 15:28:08 +01:00
Yonghong Song
ea72b0319d BPF: make 32bit register spill with 64bit alignment
In llvm, for non-alu32 mode, the stack alignment is 64bit so only one
64bit spill per 64bit slot. For alu32 mode, the stack alignment
is 32bit, so it is possible to have two 32bit spills per
64bit slot.

Currently, bpf kernel verifier does not preserve register states
for 32bit spills. That is, one 32bit register may hold a constant
value or a bounded range before spill. After reload from the
stack, the information is lost and sometimes this may cause
verifier failure. For 64bit register spill, the verifier
indeed tries to preserve the register state for reloading.

The current verifier can be modestly changed to handle one
32bit spill per 64bit stack slot with state-preserving reload.
Handling two 32bit spills per 64bit stack slot will require
substantial changes.

This patch changes stack alignment for alu32 to be 64bit.
This way, for any 64bit slot in alu32 mode, only one
32bit or 64bit register values can be saved. Together
with previous-mentioned verifier enhancement, 32bit
spill can be handled with state preserving.

Note that llvm stack slot coallescing
seems only doing adjacent packing which may leave some holes
in the stack. For example,
   stack slot 8   <== 8 bytes
   stack slot 4   <== 8 bytes with 4 byte hole
   stack slot 8   <== 8 bytes
   stack slot 4   <== 4 bytes

Differential Revision: https://reviews.llvm.org/D109073
2021-09-20 21:00:25 -07:00
Yonghong Song
e52617c31d BPF: change BTF_KIND_TAG format
Previously we have the following binary representation:
    struct bpf_type { name, info, type }
    struct btf_tag { __u32 component_idx; }
If the tag points to a struct/union/var/func type, we will have
   kflag = 1, component_idx = 0
if the tag points to struct/union member or func argument, we will have
   kflag = 0, component_idx = 0, ..., vlen - 1

The above rather makes interface complex to have both kflag and
component needed to determine its legality and index.

This patch simplifies the interface by removing kflag involvement.
   component_idx = (u32)-1 : tag pointing to a type
   component_idx = 0 ... vlen - 1 : tag pointing to a member or argument
and kflag is always 0 and there is no need to check.

Differential Revision: https://reviews.llvm.org/D109560
2021-09-09 19:03:57 -07:00
Peter Smith
e63455d5e0 [MC] Use local MCSubtargetInfo in writeNops
On some architectures such as Arm and X86 the encoding for a nop may
change depending on the subtarget in operation at the time of
encoding. This change replaces the per module MCSubtargetInfo retained
by the targets AsmBackend in favour of passing through the local
MCSubtargetInfo in operation at the time.

On Arm using the architectural NOP instruction can have a performance
benefit on some implementations.

For Arm I've deleted the copy of the AsmBackend's MCSubtargetInfo to
limit the chances of this causing problems in the future. I've not
done this for other targets such as X86 as there is more frequent use
of the MCSubtargetInfo and it looks to be for stable properties that
we would not expect to vary per function.

This change required threading STI through MCNopsFragment and
MCBoundaryAlignFragment.

I've attempted to take into account the in tree experimental backends.

Differential Revision: https://reviews.llvm.org/D45962
2021-09-07 15:46:19 +01:00
Yonghong Song
4948927058 [BPF] support btf_tag attribute in .BTF section
A new kind BTF_KIND_TAG is added to .BTF to encode
btf_tag attributes. The format looks like
   CommonType.name : attribute string
   CommonType.type : attached to a struct/union/func/var.
   CommonType.info : encoding BTF_KIND_TAG
                     kflag == 1 to indicate the attribute is
                     for CommonType.type, or kflag == 0
                     for struct/union member or func argument.
   one uint32_t    : to encode which member/argument starting from 0.

If one particular type or member/argument has more than one attribute,
multiple BTF_KIND_TAG will be generated.

Differential Revision: https://reviews.llvm.org/D106622
2021-08-28 21:02:27 -07:00
Yonghong Song
e52946b9ab BPF: avoid NE/EQ loop exit condition
Kuniyuki Iwashima reported in [1] that llvm compiler may
convert a loop exit condition with "i < bound" to "i != bound", where
"i" is the loop index variable and "bound" is the upper bound.
In case that "bound" is not a constant, verifier will always have "i != bound"
true, which will cause verifier failure since to verifier this is
an infinite loop.

The fix is to avoid transforming "i < bound" to "i != bound".
In llvm, the transformation is done by IndVarSimplify pass.
The compiler checks loop condition cost (i = i + 1) and if the
cost is lower, it may transform "i < bound" to "i != bound".
This patch implemented getArithmeticInstrCost() in BPF TargetTransformInfo
class to return a higher cost for such an operation, which
will prevent the transformation for the test case
added in this patch.

 [1] https://lore.kernel.org/netdev/1994df05-8f01-371f-3c3b-d33d7836878c@fb.com/

Differential Revision: https://reviews.llvm.org/D107483
2021-08-04 16:54:16 -07:00