704 Commits

Author SHA1 Message Date
yonghong-song
c3fb2e1cee
[BPF] Support Jump Table (#149715)
Add jump table (switch statement and computed goto) support for BPF
backend.
A `gotox <reg>` insn is implemented and the `<reg>` holds the target
insn where the gotox will go.

For a switch statement like
```
...
            switch (ctx->x) {
            case 1: ret_user = 18; break;
            case 20: ret_user = 6; break;
            case 16: ret_user = 9; break;
            case 6: ret_user = 16; break;
            case 8: ret_user = 14; break;
            case 30: ret_user = 2; break;
            default: ret_user = 1; break;
            }
...
```
and the final binary
```
    The final binary:
       4:       67 01 00 00 03 00 00 00 r1 <<= 0x3                                                           
       5:       18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0x0 ll                                  
                0000000000000028:  R_BPF_64_64  BPF.JT.0.0                                                   
       7:       0f 12 00 00 00 00 00 00 r2 += r1
        ...
    Symbol table:
     4: 0000000000000000   240 OBJECT  GLOBAL DEFAULT     4 BPF.JT.0.0
     5: 0000000000000000     4 OBJECT  GLOBAL DEFAULT     6 ret_user
     6: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   UND bar
     7: 00000000000000f0   256 OBJECT  GLOBAL DEFAULT     4 BPF.JT.0.1
    and
  [ 4] .jumptables       PROGBITS        0000000000000000 0001c8 0001f0 00      0   0  1
```
Note that for the above example, `-mllvm -bpf-min-jump-table-entries=5`
should be in compilation flags as the current default
bpf-min-jump-table-entries is 13. For example.
```
clang --target=bpf -mcpu=v4 -O2 -mllvm -bpf-min-jump-table-entries=5 -S -g test.c
```

For computed goto like
```
      int foo(int a, int b) {
        __label__ l1, l2, l3, l4;
        void *jt1[] = {[0]=&&l1, [1]=&&l2};
        void *jt2[] = {[0]=&&l3, [1]=&&l4};
        int ret = 0;
    
        goto *jt1[a % 2];
        l1: ret += 1;
        l2: ret += 3;
        goto *jt2[b % 2];
        l3: ret += 5;
        l4: ret += 7;
        return ret;
      }
```
The final binary:
```
      12:       bf 23 20 00 00 00 00 00 r3 = (s32)r2
      13:       67 03 00 00 03 00 00 00 r3 <<= 0x3
      14:       18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0x0 ll
                0000000000000070:  R_BPF_64_64  BPF.JT.0.0
      16:       0f 32 00 00 00 00 00 00 r2 += r3
      17:       bf 11 20 00 00 00 00 00 r1 = (s32)r1
      18:       67 01 00 00 03 00 00 00 r1 <<= 0x3
      19:       18 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r3 = 0x0 ll
                0000000000000098:  R_BPF_64_64  BPF.JT.0.1
      21:       0f 13 00 00 00 00 00 00 r3 += r1
    
  [ 4] .jumptables       PROGBITS        0000000000000000 000160 000020 00      0   0  1

     4: 0000000000000000    16 OBJECT  GLOBAL DEFAULT     4 BPF.JT.0.0
     5: 0000000000000010    16 OBJECT  GLOBAL DEFAULT     4 BPF.JT.0.1
```

A more complicated test with both switch-statement triggered jump table
and compute gotos:

```
$ cat test3.c
struct simple_ctx {
        int x;
        int y;
        int z;
};

int ret_user, ret_user2;
void bar(void);
int foo(struct simple_ctx *ctx, struct simple_ctx *ctx2, int a, int b)
{
        __label__ l1, l2, l3, l4;
        void *jt1[] = {[0]=&&l1, [1]=&&l2};
        void *jt2[] = {[0]=&&l3, [1]=&&l4};
        int ret = 0;

        goto *jt1[a % 2];
        l1: ret += 1;
        l2: ret += 3;
        goto *jt2[b % 2];
        l3: ret += 5;
        l4: ret += 7;

        bar();

        switch (ctx->x) {
        case 1: ret_user = 18; break;
        case 20: ret_user = 6; break;
        case 16: ret_user = 9; break;
        case 6: ret_user = 16; break;
        case 8: ret_user = 14; break;
        case 30: ret_user = 2; break;
        default: ret_user = 1; break;
        }

        return ret;
}
```
Compile with
```
  clang --target=bpf -mcpu=v4 -O2 -S test3.c
  clang --target=bpf -mcpu=v4 -O2 -c test3.c
```
  The binary:
```
     /* For computed goto */
      13:       bf 42 20 00 00 00 00 00 r2 = (s32)r4                                                         
      14:       67 02 00 00 03 00 00 00 r2 <<= 0x3                                                           
      15:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x0 ll                                  
                0000000000000078:  R_BPF_64_64  BPF.JT.0.1                                                   
      17:       0f 21 00 00 00 00 00 00 r1 += r2                                                             
      18:       bf 32 20 00 00 00 00 00 r2 = (s32)r3                                                         
      19:       67 02 00 00 03 00 00 00 r2 <<= 0x3                                                           
      20:       18 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r3 = 0x0 ll                                  
                00000000000000a0:  R_BPF_64_64  BPF.JT.0.2                                                   
      22:       0f 23 00 00 00 00 00 00 r3 += r2

      /* For switch statement */
      39:       67 01 00 00 03 00 00 00 r1 <<= 0x3
      40:       18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0x0 ll
                0000000000000140:  R_BPF_64_64  BPF.JT.0.0
      42:       0f 12 00 00 00 00 00 00 r2 += r1
```
You can see jump table symbols are all different.
2025-09-16 09:27:08 -07:00
Rahul Joshi
7b6db769c3
[NFC][DecoderEmitter] Code cleanup in DecoderEmitter::emitTable (#158014)
Several code cleanup changes in code to emit decoder tables:
- Start comments on each line at a fixed column for readibility.
- Combine repeated code to decode and emit ULEB128 into a single
function.
- Add helper `getDecoderOpName` to print decoder op.
- Print Filter/CheckField/predicate index values with those opcodes.
2025-09-15 17:51:29 -07:00
Reid Kleckner
f3efbce4a7
[llvm] Move data layout string computation to TargetParser (#157612)
Clang and other frontends generally need the LLVM data layout string in
order to generate LLVM IR modules for LLVM. MLIR clients often need it
as well, since MLIR users often lower to LLVM IR.

Before this change, the LLVM datalayout string was computed in the
LLVM${TGT}CodeGen library in the relevant TargetMachine subclass.
However, none of the logic for computing the data layout string requires
any details of code generation. Clients who want to avoid duplicating
this information were forced to link in LLVMCodeGen and all registered
targets, leading to bloated binaries. This happened in PR #145899,
which measurably increased binary size for some of our users.

By moving this information to the TargetParser library, we
can delete the duplicate datalayout strings in Clang, and retain the
ability to generate IR for unregistered targets.

This is intended to be a very mechanical LLVM-only change, but there is
an immediately obvious follow-up to clang, which will be prepared
separately.

The vast majority of data layouts are computable with two inputs: the
triple and the "ABI name". There is only one exception, NVPTX, which has
a cl::opt to enable short device pointers. I invented a "shortptr" ABI
name to pass this option through the target independent interface.
Everything else fits. Mips is a bit awkward because it uses a special
MipsABIInfo abstraction, which includes members with codegen-like
concepts like ABI physical registers that can't live in TargetParser. I
think the string logic of looking for "n32" "n64" etc is reasonable to
duplicate. We have plenty of other minor duplication to preserve
layering.

---------

Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Co-authored-by: Sergei Barannikov <barannikov88@gmail.com>
2025-09-11 11:05:29 -07:00
Sergei Barannikov
69f1aebf20
[BPF] Remove skb operand of LD_ABS/LD_IND instructions (#157344)
The instructions already have R6 register in the Uses list, there is no
need for an additional explicit `GPR:$skb` operand.

This simplifies intrinsic selection and makes the instructions decodable
without post-decoding pass inserting R6 operand.
2025-09-09 02:20:00 +03:00
Matt Arsenault
727e9f5ea5
CodeGen: Pass SubtargetInfo to TargetGenInstrInfo constructors (#157337)
This will make it possible for tablegen to make subtarget
dependent decisions without adding new arguments to every
target.

---------

Co-authored-by: Sergei Barannikov <barannikov88@gmail.com>
2025-09-08 12:12:19 +09:00
Kazu Hirata
0dbd39afa2
[BPF] Remove an unnecessary cast (NFC) (#157230)
getZExtValue() already returns uint64_t.
2025-09-06 09:51:59 -07:00
Sergei Barannikov
ed3597e2f7
[TableGen][Decoder] Decode operands with zero width or all bits known (#156358)
There are two classes of operands that DecoderEmitter cannot currently
handle:
1. Operands that do not participate in instruction encoding.
2. Operands whose encoding contains only 1s and 0s.

Because of this, targets developed various workarounds. Some targets
insert missing operands after an instruction has been (incompletely)
decoded, other take into account the missing operands when printing the
instruction. Some targets do neither of that and fail to correctly
disassemble some instructions.

This patch makes it possible to decode both classes of operands and
allows to remove existing workarounds.

For the case of operand with no contribution to instruction encoding,
one should now add `bits<0> OpName` field to instruction encoding
record. This will make DecoderEmitter generate a call to the decoder
function specified by the operand's DecoderMethod. The function has a
signature different from the usual one and looks like this:

```
static DecodeStatus DecodeImm42Operand(MCInst &Inst, const MCDisassembler *Decoder) {
  Inst.addOperand(MCOperand::createImm(42));
  return DecodeStatus::Success;
}
```

Notably, encoding bits are not passed to it (since there are none).

There is nothing special about the second case, the operand bits are
passed as usual. The difference is that before this change, the function
was not called if all the bits of the operand were known (no '?' in the
operand encoding).

There are two options controlling the behavior. Passing an option
enables the old behavior. They exist to allow smooth transition to the
new behavior. They are temporary (yeah, I know) and will be removed once
all targets migrate, possibly giving some more time to downstream
targets.

Subsequent patches in the stack enable the new behavior on some in-tree
targets.
2025-09-04 14:48:36 +00:00
Rahul Joshi
22f8693248
[NFC][MC][Decoder] Extract fixed pieces of decoder code into new header file (#154802)
Extract fixed functions generated by decoder emitter into a new
MCDecoder.h header.
2025-08-21 15:06:43 -07:00
Sergei Barannikov
d6679d5a5f
[Target] Remove SoftFail field on targets that don't use it (NFC) (#154659)
That is, on all targets except ARM and AArch64.
This field used to be required due to a bug, it was fixed long ago
by 23423c0ea8d414e56081cb6a13bd8b2cc91513a9.
2025-08-21 05:21:42 +03:00
Fangrui Song
d3589edafc MCAsmBackend::applyFixup: Change Data to indicate the relocated location
`Data` now references the first byte of the fixup offset within the current fragment.

MCAssembler::layout asserts that the fixup offset is within either the
fixed-size content or the optional variable-size tail, as this is the
most the generic code can validate without knowing the target-specific
fixup size.

Many backends applyFixup assert
```
assert(Offset + Size <= F.getSize() && "Invalid fixup offset!");
```

This refactoring allows a subsequent change to move the fixed-size
content outside of MCSection::ContentStorage, fixing the
-fsanitize=pointer-overflow issue of #150846

Pull Request: https://github.com/llvm/llvm-project/pull/151724
2025-08-02 09:27:06 -07:00
Mahé
379949d79f
[BPF] Visit nested map array during BTF generation (#150608)
Fixes missing inner map struct type definitions [^1]. We should visit
the type of nested array of maps like we do for global maps. This patch
adds a boolean to convey the information to visitTypeEntry and
visitDerivedType that the pointee is a map definition and should be
treated as such.

It ressembles and works with commit 0d21c956a5c1 ("[BPF] Handle nested
wrapper structs in BPF map definition traversal (#144097)") which
focused on directly nested wrapper structs.

Before that patch, this ARRAY_OF_MAPS definition would lead to the BTF
information include the 'missing_type' as "FWD 'missing_type'
fwd_kind=struct":

	struct missing_type { uint64_t foo; };
	struct {
		__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
		[...]
		__array(
			values, struct {
				[...]
				__type(value, struct missing_type);
			});
	} map SEC(".maps");

Which lead to errors while trying to load the map:

libbpf: map 'outer_map.inner': can't determine value size for type [N]:
-22.

To solve this issue, users had to use the struct in a dummy variable or
in a dummy function for the BTF to be generated correctly [^2].

[^1]: https://lore.kernel.org/netdev/aH_cGvgC20iD8qs9@gmail.com/T/#u
[^2]:
https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339

---------

Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Co-authored-by: Eduard Zingerman <eddyz87@gmail.com>
2025-07-28 13:45:15 -07:00
Fangrui Song
d9489fd073 AVR,BPF: Derive from MCAsmInfoELF
instead of MCAsmInfo. MCAsmInfo is derived by object file format
MCAsmInfo.
2025-07-26 14:42:08 -07:00
Fangrui Song
2571924ad6 MCSectionELF: Remove classof
The object file format specific derived classes are used in context like
MCStreamer and MCObjectTargetWriter where the type is statically known.
We don't use isa/dyn_cast and we want to eliminate
MCSection::SectionVariant in the base class.
2025-07-25 09:50:19 -07:00
Kazu Hirata
3e53d4d386
[llvm] Remove unused includes (NFC) (#150265)
These are identified by misc-include-cleaner.  I've filtered out those
that break builds.  Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
2025-07-23 15:18:46 -07:00
yonghong-song
5ca40fa101
[BPF] Fix build warnings due to a static var in header file (#150128)
Simon Pilgrim ([1]) and Anton reported that the following warning will
appear when building clang compiler:
```
In file included from .../llvm-project/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp:9: .../llvm-project/llvm/lib/Target/BPF/BPF.h:25:20: warning: ‘llvm::BPF_TRAP’ defined but not used [-Wunused-variable]
   25 | static const char *BPF_TRAP = "__bpf_trap";
      |                    ^~~~~~~~
...
In file included from .../llvm-project/llvm/lib/Target/BPF/MCTargetDesc/BPFInstPrinter.cpp:14:
.../llvm-project/llvm/lib/Target/BPF/BPF.h:25:20: warning: ‘llvm::BPF_TRAP’ defined but not used [-Wunused-variable]
   25 | static const char *BPF_TRAP = "__bpf_trap";
      |                    ^~~~~~~~
...
```
Instead of using static const variable, use macro to silence warnings.

  [1] https://github.com/llvm/llvm-project/pull/131731
2025-07-22 16:37:10 -07:00
Fangrui Song
5ba458c559 MCFixup: Replace getTargetKind with getKind 2025-07-15 00:21:07 -07:00
Boyao Wang
697beb3f17
[TargetLowering] Change getOptimalMemOpType and findOptimalMemOpLowering to take LLVM Context (#147664)
Add LLVM Context to getOptimalMemOpType and findOptimalMemOpLowering. So
that we can use EVT::getVectorVT to generate EVT type in
getOptimalMemOpType.

Related to [#146673](https://github.com/llvm/llvm-project/pull/146673).
2025-07-10 11:11:09 +08:00
Fangrui Song
244e053b6c MC: Remove llvm/MC/MCFixupKindInfo.h
The file used to define `MCFixupKindInfo`, a simple structure,
which is now in MCAsmBackend.h.
2025-07-05 11:24:11 -07:00
Fangrui Song
a26f8f2348 BPF: Replace deprecated FK_PCRel_ with FK_Data_ fixup and PCRel flag
We will unify the generic fixup kinds FK_Data_ and FK_PCRel_. A
FK_PCRel_ kind is essentially the corresponding FK_Data_ fixup with the
PCRel flag set.
2025-07-04 22:30:23 -07:00
Fangrui Song
21cf7bd4ec BPFMCCodeEmitter: Set PCRel at fixup creation
Avoid reliance on the MCAssembler::evaluateFixup workaround that checks
MCFixupKindInfo::FKF_IsPCRel. Additionally, standardize how fixups are
appended. This helper will facilitate future fixup data structure
optimizations.
2025-07-04 17:52:43 -07:00
Fangrui Song
dd2891535d
MCAsmBackend: Merge addReloc into applyFixup (#146820)
Follow-up to #141333. Relocation generation called both addReloc and
applyFixup, with the default addReloc invoking shouldForceRelocation,
resulting in three virtual calls. This approach was also inflexible, as
targets needing additional data required extending
`shouldForceRelocation` (see #73721, resolved by #141311).

This change integrates relocation handling into applyFixup, eliminating
two virtual calls. The prior default addReloc is renamed to
maybeAddReloc. Targets overriding addReloc now call their customized
addReloc implementation.
2025-07-02 23:14:11 -07:00
yonghong-song
13fddeaf2c
[BPF] Emit proper error message for insns with tied operands (#146778)
Jonathan Cottrill reported a crash in [1] with the following command
line:
```
  $ echo 'r0 = atomic_fetch_add((u64*)(r2 + 0), r1)' | llvm-mc --arch bpf --filetype null
```

Note that in the above command, the insn specification requires that r0
and r1 must be the same register. Otherwise, the crash will happen.

Let us add a case Match_InvalidTiedOperand to handle such invalid insns.
With this patch, the error message looks like below:
```
  <stdin>:1:39: error: operand is not the same as the dst register
  r0 = atomic_fetch_add((u64*)(r2 + 0), r1)
                                        ^
```
The error message is much better than the crash. Some other insns are
also covered by this patch.
```
  $ echo 'w0 = xchg32_32(r2 + 0, w1)' | llvm-mc --arch bpf --filetype null
  <stdin>:1:24: error: operand is not the same as the dst register
  w0 = xchg32_32(r2 + 0, w1)
                         ^
```
  [1] https://github.com/llvm/llvm-project/issues/145180

Co-authored-by: Yonghong Song <yonghong.song@linux.dev>
2025-07-02 18:28:05 -07:00
Fangrui Song
f96832a520 BPF: Replace deprecated MCExpr::print with MCAsmInfo::printExpr 2025-06-28 12:27:23 -07:00
Fangrui Song
e878b7e349 MCParsedAsmOperand::print: Add MCAsmInfo parameter
so that subclasses can provide the appropriate MCAsmInfo to print
MCExpr objects.

At present, llvm/utils/TableGen/AsmMatcherEmitter.cpp constucts a
generic MCAsmInfo.
2025-06-28 12:05:33 -07:00
Fangrui Song
d93aff42c2 MC: Migrate away from operator<< MCExpr
MCExpr::print has an optional MCAsmInfo argument, which is error-prone
when omitted. MCExpr::print and the convenience helper operator<< are
discouraged to use. Switch to MCAsmInfo::printExpr instead. Use the
target-specific MCAsmInfo if available.
2025-06-28 10:58:09 -07:00
Fangrui Song
af2bf2e544 MC: Remove unneeded MCSymbolRefExpr::VariantKind calls 2025-06-27 09:47:53 -07:00
Michal Rostecki
0d21c956a5
[BPF] Handle nested wrapper structs in BPF map definition traversal (#144097)
In Aya/Rust, BPF map definitions are nested in two nested types:

* A struct representing the map type (e.g., `HashMap`, `RingBuf`) that
provides methods for interacting with the map type (e.g. `HashMap::get`,
`RingBuf::reserve`).
* An `UnsafeCell`, which informs the Rust compiler that the type is
thread-safe and can be safely mutated even as a global variable. The
kernel guarantees map operation safety.

This leads to a type hierarchy like:

```rust
    pub struct HashMap<K, V, const M: usize, const F: usize = 0>(
        core::cell::UnsafeCell<HashMapDef<K, V, M, F>>,
    );
    const BPF_MAP_TYPE_HASH: usize = 1;
    pub struct HashMapDef<K, V, const M: usize, const F: usize = 0> {
        r#type: *const [i32; BPF_MAP_TYPE_HASH],
        key: *const K,
        value: *const V,
        max_entries: *const [i32; M],
        map_flags: *const [i32; F],
    }
```

Then used in the BPF program code as a global variable:

```rust
    #[link_section = ".maps"]
    static HASH_MAP: HashMap<u32, u32, 1337> = HashMap::new();
```

Which is an equivalent of the following BPF map definition in C:

```c
    #define BPF_MAP_TYPE_HASH 1
    struct {
        int (*type)[BPF_MAP_TYPE_HASH];
        typeof(int) *key;
        typeof(int) *value;
        int (*max_entries)[1337];
    } map_1 __attribute__((section(".maps")));
```

Accessing the actual map definition requires traversing:

```
  HASH_MAP -> __0 -> value
```

Previously, the BPF backend only visited the pointee types of the
outermost struct, and didn’t descend into inner wrappers. This caused
issues when the key/value types were custom structs:

```rust
    // Define custom structs for key and values.
    pub struct MyKey(u32);
    pub struct MyValue(u32);

    #[link_section = ".maps"]
    #[export_name = "HASH_MAP"]
    pub static HASH_MAP: HashMap<MyKey, MyValue, 10> = HashMap::new();
```

These types weren’t fully visited and appeared in BTF as forward
declarations:

```
    #30: <FWD> 'MyKey' kind:struct
    #31: <FWD> 'MyValue' kind:struct
```

The fix is to enhance `visitMapDefType` to recursively visit inner
composite members. If a member is a composite type (likely a wrapper),
it is now also visited using `visitMapDefType`, ensuring that the
pointee types of the innermost stuct members, like `MyKey` and
`MyValue`, are fully resolved in BTF.

With this fix, the correct BTF entries are emitted:

```
    #6: <STRUCT> 'MyKey' sz:4 n:1
            #00 '__0' off:0 --> [7]
    #7: <INT> 'u32' bits:32 off:0
    #8: <PTR> --> [9]
    #9: <STRUCT> 'MyValue' sz:4 n:1
            #00 '__0' off:0 --> [7]
```

Fixes: #143361
2025-06-20 10:17:36 -07:00
Kazu Hirata
a8edda195c
[llvm] Remove unused includes (NFC) (#144941)
These are identified by misc-include-cleaner.  I've filtered out those
that break builds.  Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
2025-06-19 14:58:05 -07:00
Andrew Rogers
19658d1474
[llvm] annotate interfaces in llvm/Target for DLL export (#143615)
## Purpose

This patch is one in a series of code-mods that annotate LLVM’s public
interface for export. This patch annotates the `llvm/Target` library.
These annotations currently have no meaningful impact on the LLVM build;
however, they are a prerequisite to support an LLVM Windows DLL (shared
library) build.

## Background

This effort is tracked in #109483. Additional context is provided in
[this
discourse](https://discourse.llvm.org/t/psa-annotating-llvm-public-interface/85307),
and documentation for `LLVM_ABI` and related annotations is found in the
LLVM repo
[here](https://github.com/llvm/llvm-project/blob/main/llvm/docs/InterfaceExportAnnotations.rst).

A sub-set of these changes were generated automatically using the
[Interface Definition Scanner (IDS)](https://github.com/compnerd/ids)
tool, followed formatting with `git clang-format`.

The bulk of this change is manual additions of `LLVM_ABI` to
`LLVMInitializeX` functions defined in .cpp files under llvm/lib/Target.
Adding `LLVM_ABI` to the function implementation is required here
because they do not `#include "llvm/Support/TargetSelect.h"`, which
contains the declarations for this functions and was already updated
with `LLVM_ABI` in a previous patch. I considered patching these files
with `#include "llvm/Support/TargetSelect.h"` instead, but since
TargetSelect.h is a large file with a bunch of preprocessor x-macro
stuff in it I was concerned it would unnecessarily impact compile times.

In addition, a number of unit tests under llvm/unittests/Target required
additional dependencies to make them build correctly against the LLVM
DLL on Windows using MSVC.

## Validation

Local builds and tests to validate cross-platform compatibility. This
included llvm, clang, and lldb on the following configurations:

- Windows with MSVC
- Windows with Clang
- Linux with GCC
- Linux with Clang
- Darwin with Clang
2025-06-17 13:28:45 -07:00
yonghong-song
5baf351ba8
[BPF] Do not allow gotol in the middle of asm insn (#144545)
Previously I accidentally allowed 'gotol' insn in the middle of asm insn
([1]). But actually 'gotol' is not allowed in the middle of any asm
insn, so remove it from isValidIdInMiddle().

[1] 6c412b6c6f
2025-06-17 09:51:40 -07:00
David Green
48e54f3a22
[CostModel] Mark all TTIImpls as final. NFC (#143404)
In the AArch64 version this helps reduce the number of blr instruction
(indirect jumps) in from 325 to 87, and reduces the size of the object
file by 4%. It seems to help make the code more efficient even if it
doesn't greatly affect compile time.

The AMDGPU variants are already marked as final.
2025-06-15 08:51:59 +01:00
yonghong-song
2f1e6eb6c3
[BPF] Report an warning if certain insn imm operand cannot fit in 32bit (#142989)
Ihor Solodrai reported a case ([1]) where gcc reports an error but clang
ignores that error and proceeds to generate incorrect code. More
specifically, the problematic code looks like:
   if r1 == 0xcafefeeddeadbeef goto <label>

Here, 0xcafefeeddeadbeef needs to be encoded in a 32-bit imm field
of the insns and the 32-bit imm allows sign extenstion to 64-bit imm.
Obviously, 0xcafefeeddeadbeef cannot encode properly.

The compilation failed for gcc with the following error:
  Error: immediate out of range, shall fit in 32 bits

Given a 64-bit imm value, converting to the proper 32-bit imm value
must satisfy the following 64-bit patterns:
  00000000 00000000 00000000 00000000 xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx
  11111111 11111111 11111111 11111111 1xxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx

So if the top 32-bits is 0 or the top 33-bits is 0x1ffffffff, then the 64-bit imm
value can be truncated into proper 32-bit imm. Otherwise, a warning
message, the same as gcc, will be issued. If -Werror is enabled during
compilation, the warning will turn into an error.

[1] https://lore.kernel.org/bpf/70affb12-327b-4882-bd1d-afda8b8c6f56@linux.dev/
2025-06-13 11:58:48 -07:00
Stephen Tozer
a08a831515
[DLCov][NFC] Propagate annotated DebugLocs through transformations (#138047)
Part of the coverage-tracking feature, following #107279.

In order for DebugLoc coverage testing to work, we firstly have to set
annotations for intentionally-empty DebugLocs, and secondly we have to
ensure that we do not drop these annotations as we propagate DebugLocs
throughout compilation. As the annotations exist as part of the DebugLoc
class, and not the underlying DILocation, they will not survive a
DebugLoc->DILocation->DebugLoc roundtrip. Therefore this patch modifies
a number of places in the compiler to propagate DebugLocs directly
rather than via the underlying DILocation. This has no effect on the
output of normal builds; it only ensures that during coverage builds, we
do not drop incorrectly annotations and therefore create false
positives.

The bulk of these changes are in replacing
DILocation::getMergedLocation(s) with a DebugLoc equivalent, and in
changing the IRBuilder to store a DebugLoc directly rather than storing
DILocations in its general Metadata array. We also use a new function,
`DebugLoc::orElse`, which selects the "best" DebugLoc out of a pair
(valid location > annotated > empty), preferring the current DebugLoc on
a tie - this encapsulates the existing behaviour at a few sites where we
_may_ assign a DebugLoc to an existing instruction, while extending the
logic to handle annotation DebugLocs at the same time.
2025-06-12 14:06:27 +01:00
Kazu Hirata
99f6cb8935
[BPF] Use heterogeneous lookup with std::map (NFC) (#143395) 2025-06-10 07:12:38 -07:00
Kazu Hirata
17999f01bb
[BPF] Initialize SmallVector (NFC) (#143392) 2025-06-09 12:45:59 -07:00
yonghong-song
cf56b536bf
[BPF] Allow struct pointee member btf generation with annotations (#141719)
Currently, to avoid generating too much BTF types, for a struct type
like
```
  struct foo {
    int val;
    struct bar *ptr;
  };
```
if the BTF generation reaches 'struct foo', it will not generate actual
type for 'struct bar' and instead a forward decl is generated. The
'struct bar' is actual generated in BTF unless it is reached through a
non-struct pointer member.

Such a limitation forces bpf developer to hack and workaround this
problem. See [1] and [2]. For example in [1], we have
```
    struct map_value {
	struct prog_test_ref_kfunc *not_kptr;
	struct prog_test_ref_kfunc __kptr *val;
	struct node_data __kptr *node;
    };
```
The BTF type for 'struct node_data' is not generated. Note that we have
a '__kptr' annotation. Similar problem for [2] with a '__uptr'
annotation. Note that the issue in [1] has been resolved later but the
hack in [2] is still needed.

This patch relaxed the struct type (with struct pointer member) BTF
generation if the struct pointer has a btf_type_tag annotation.

[1] https://lore.kernel.org/r/20230310230743.2320707-4-davemarchevsky@fb.com
[2] https://lore.kernel.org/r/20241023234759.860539-9-martin.lau@linux.dev
2025-05-28 14:36:10 -07:00
Justin Bogner
b7bb256703
Warn on misuse of DiagnosticInfo classes that hold Twines (#137397)
This annotates the `Twine` passed to the constructors of the various
DiagnosticInfo subclasses with `[[clang::lifetimebound]]`, which causes
us to warn when we would try to print the twine after it had already
been destructed.

We also update `DiagnosticInfoUnsupported` to hold a `const Twine &`
like all of the other DiagnosticInfo classes, since this warning allows
us to clean up all of the places where it was being used incorrectly.
2025-05-28 12:26:39 -07:00
yonghong-song
ab391beb11
[BPF] Handle traps with kfunc call __bpf_trap (#131731)
Currently, middle-end generates 'unreachable' insn if the compiler
feels the code is indeed unreachable or the code becomes invalid
due to some optimizaiton (e.g. code optimization with uninitialized
variables).

Right now BPF backend ignores 'unreachable' insn during selectiondag
lowering. For cases where 'unreachable' is due to invalid code
transformation, such a signal will be missed. Later on, users needs
some effort to debug it which impacts developer productivity.

This patch enabled selectiondag lowering for 'unreachable' insn.

Previous attempt ([1]) tries to have a backend IR pass to filter
out 'unreachable' insns in a number of cases. But such pattern
matching may misalign with future middle-end optimization with
'unreachable' insns.

This patch takes a different approach. The 'unreachable' insn is
lowered with special encoding in bpf object file and verifier
will do proper verification for the bpf prog. More specifically,
the 'unreachable' insn is replaced by a __bpf_trap() function.
This function will be a kfunc (in ".ksyms" section) with a weak
attribute, but does not have definition. The actual kfunc definition
is expected to be in kernel. The __bpf_trap() extern function
is also encoded in BTF. The name __bpf_trap() is chosen to satisfy
reserved identifier requirement.

Besides the uninitialized variable case, the builtin function
'__builtin_trap' can also generate kfunc __bpf_trap(). For example
in [3], we have
```
  # define __bpf_unreachable()  __builtin_trap()
```
If the compiler didn't remove __builtin_trap() during middle-end
optimization, compilation will fail.

With this patch, compilation will not fail and __builtin_trap()
is converted to __bpf_trap() kfunc. The eventual failure will be
in verifier instead of llvm compilation. To keep compilation
time failure, user can add an option like `-ftrap-function=<something>`.

I tested this patch on bpf selftests and all tests are passed.
I also tried original example in [2] and the code looks like below:
```
          ; {
               0:       bf 16 00 00 00 00 00 00 r6 = r1
          ;       bpf_printk("Start");
               1:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x0 ll
                        0000000000000008:  R_BPF_64_64  .rodata
               3:       b4 02 00 00 06 00 00 00 w2 = 0x6
               4:       85 00 00 00 06 00 00 00 call 0x6
          ; DEFINE_FUNC_CTX_POINTER(data)
               5:       61 61 4c 00 00 00 00 00 w1 = *(u32 *)(r6 + 0x4c)
          ;       bpf_printk("pre ipv6_hdrlen_offset");
               6:       18 01 00 00 06 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x6 ll
                        0000000000000030:  R_BPF_64_64  .rodata
               8:       b4 02 00 00 17 00 00 00 w2 = 0x17
               9:       85 00 00 00 06 00 00 00 call 0x6
              10:       85 10 00 00 ff ff ff ff call -0x1
                        0000000000000050:  R_BPF_64_32  __bpf_trap
              11:       95 00 00 00 00 00 00 00 exit
          <END>
```
Eventually kernel verifier will emit the following logs:
```
      10: (85) call __bpf_trap#74479
      unexpected __bpf_trap() due to uninitialized variable?
```
In another internal sched-ext bpf prog, with the patch we have bpf code:
```
  Disassembly of section .text:
  0000000000000000 <scx_storage_init_single>:
  ; {
       0:       bc 13 00 00 00 00 00 00 w3 = w1
       1:       b4 01 00 00 00 00 00 00 w1 = 0x0
  ;       const u32 zero = 0;
  ...
  0000000000003a80 <create_dom>:
  ; {
    1872:       bc 16 00 00 00 00 00 00 w6 = w1
  ;       bpf_printk("dom_id %d", dom_id);
    1873:       18 01 00 00 3f 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x3f ll
                0000000000003a88:  R_BPF_64_64  .rodata
    1875:       b4 02 00 00 0a 00 00 00 w2 = 0xa
    1876:       bc 63 00 00 00 00 00 00 w3 = w6
    1877:       85 00 00 00 06 00 00 00 call 0x6
  ;       ret = scx_bpf_create_dsq(dom_id, 0);
    1878:       bc 61 00 00 00 00 00 00 w1 = w6
    1879:       b4 02 00 00 00 00 00 00 w2 = 0x0
    1880:       85 10 00 00 ff ff ff ff call -0x1
                0000000000003ac0:  R_BPF_64_32  scx_bpf_create_dsq
  ;       domc->node_cpumask = node_data[node_id];
    1881:       85 10 00 00 ff ff ff ff call -0x1
                0000000000003ac8:  R_BPF_64_32  __bpf_trap
    1882:       95 00 00 00 00 00 00 00 exit
    <END>
```
The verifier can easily report the error too.

A bpf flag `-bpf-disable-trap-unreachable` is introduced to disable
trapping for 'unreachable' or __builtin_trap.

  [1] https://github.com/llvm/llvm-project/pull/126858
  [2] https://github.com/msune/clang_bpf/blob/main/Makefile#L3
  [3] https://github.com/libbpf/libbpf/blob/master/src/bpf_helpers.h
2025-05-27 13:34:15 -07:00
Fangrui Song
a0901a2f87 Replace #include MCAsmLexer.h with AsmLexer.h
MCAsmLexer.h has been made a forwarder header since #134207
2025-05-25 11:57:29 -07:00
Fangrui Song
fe32806d67 ELFObjectWriter: Remove the MCContext argument from getRelocType
Additionally, swap MCFixup/MCValue order to match addReloc/recordRelocation.
2025-05-24 21:48:11 -07:00
Fangrui Song
871b0a3221
MCAsmBackend: Simplify applyFixup (#141333)
Remove the MCSubtargetInfo argument from applyFixup, introduced in
https://reviews.llvm.org/D45962 , as it's only required by ARM. Instead,
add const MCFragment & so that ARMAsmBackend can retrieve
MCSubtargetInfo via a static member function.

Additionally, remove the MCAssembler argument, which is also only
required by ARM.

Additionally, make applyReloc non-const. Its arguments now fully cover
addReloc's functionality.
2025-05-23 23:09:56 -07:00
Jason Eckhardt
d56f23e408
[AsmParser] Replace starIsStartOfStatement with tokenIsStartOfStatement. (#137997)
Currently `MCTargetAsmParser::starIsStartOfStatement` checks for `*` at
the start of the statement. There are other (currently) downstream
back-ends that need the same treatment for other tokens. Instead of
introducing bespoke APIs for each such token, we generalize (and rename)
starIsStartOfStatement as tokenIsStartOfStatement which takes the token
of interest as an argument.

Update the BPF AsmParser (the only upstream consumer today) to use the
new version.
2025-05-07 12:36:17 -05:00
Matthias Braun
675cb70641
Register assembly printer passes (#138348)
Register assembly printer passes in the pass registry.

This makes it possible to use `llc -start-before=<target>-asm-printer ...` in tests.

Adds a `char &ID` parameter to the AssemblyPrinter constructor to allow
targets to use the `INITIALIZE_PASS` macros and register the pass in the
pass registry. This currently has a default parameter so it won't break
any targets that have not been updated.
2025-05-06 18:01:17 -07:00
Kazu Hirata
aa15596b5f
[llvm] Remove unused local variables (NFC) (#138478) 2025-05-04 21:33:54 -07:00
Craig Topper
ca21508080
[Targets] Migrate from atomic_load_8/16/32/64 to atomic_load_nonext_8/16/32/64. NFC (#137428)
This makes them more consistent with the checks performed by regular loads. We can't simply add IsNonExtLoad to the existing atomic_load_8/16/32/64 as that would affect out of tree targets.
2025-04-28 09:26:34 -07:00
Sergei Barannikov
bb1765179e
[TTI] Simplify implementation (NFCI) (#136674)
Replace "concept based polymorphism" with simpler PImpl idiom.

This pursues two goals:
* Enforce static type checking. Previously, target implementations hid
base class methods and type checking was impossible. Now that they
override the methods, the compiler will complain on mismatched
signatures.
* Make the code easier to navigate. Previously, if you asked your
favorite LSP server to show a method (e.g. `getInstructionCost()`), it
would show you methods from `TTI`, `TTI::Concept`, `TTI::Model`,
`TTIImplBase`, and target overrides. Now it is two less :)

There are three commits to hopefully simplify the review.

The first commit removes `TTI::Model`. This is done by deriving
`TargetTransformInfoImplBase` from `TTI::Concept`. This is possible
because they implement the same set of interfaces with identical
signatures.

The first commit makes `TargetTransformImplBase` polymorphic, which
means all derived classes should `override` its methods. This is done in
second commit to make the first one smaller. It appeared infeasible to
extract this into a separate PR because the first commit landed
separately would result in tons of `-Woverloaded-virtual` warnings (and
break `-Werror` builds).

The third commit eliminates `TTI::Concept` by merging it with the only
derived class `TargetTransformImplBase`. This commit could be extracted
into a separate PR, but it touches the same lines in
`TargetTransformInfoImpl.h` (removes `override` added by the second
commit and adds `virtual`), so I thought it may make sense to land these
two commits together.

Pull Request: https://github.com/llvm/llvm-project/pull/136674
2025-04-26 15:25:40 +03:00
Craig Topper
2ca071b1de
[TableGen][RISCV][AArch64][GISel] Properly implement isAnyExtLoad/isSignExtLoad/isZeroExtLoad for IsAtomic in SelectionDAG. (#137096)
Support isAnyExtLoad() for IsAtomic in GISel.

Modify atomic_load_az* to check for extload or zextload. And rename to
atomic_load_azext*

Add atomic_load_asext* and use in RISC-V. I used "asext" rather than
"as" so it wouldn't be confused with the word "as".
2025-04-24 08:27:38 -07:00
Simon Pilgrim
a99e055030
[DAG] shouldReduceLoadWidth - add optional<unsigned> byte offset argument (#136723)
Based off feedback for #129695 - we need to be able to determine the
load offset of smaller loads when trying to determine whether a multiple
use load should be split (in particular for AVX subvector extractions).

This patch adds a std::optional<unsigned> ByteOffset argument to
shouldReduceLoadWidth calls for where we know the constant offset to
allow targets to make use of it in future patches.
2025-04-23 12:30:27 +01:00
Sergei Barannikov
3334c3597d
[TTI] Fix discrepancies in prototypes between interface and implementations (NFCI) (#136655)
These are not diagnosed because implementations hide the methods of the base class rather than overriding them.
This works as long as a hiding function is callable with the same arguments as the same function from the base class.

Pull Request: https://github.com/llvm/llvm-project/pull/136655
2025-04-22 11:40:12 +03:00
Sergei Barannikov
0014b49482
[TTI] Make all interface methods const (NFCI) (#136598)
Making `TargetTransformInfo::Model::Impl` `const` makes sure all
interface methods are `const`, in `BasicTTIImpl`, its bases, and in all
derived classes.

Pull Request: https://github.com/llvm/llvm-project/pull/136598
2025-04-22 06:27:29 +03:00