llvm-project

History

[X86] Reduce znver3/4 LoopMicroOpBufferSize to practical loop unrolling values (#91340 )

The znver3/4 scheduler models have previously associated the LoopMicroOpBufferSize with the maximum size of their op caches, and when this led to quadratic complexity issues this were reduced to a value of 512 uops, based mainly on compilation time and not its effectiveness on runtime performance.

From a runtime performance POV, a large LoopMicroOpBufferSize leads to a higher number of loop unrolls, meaning the cpu has to rely on the frontend decode rate (4 ins/cy max) for much longer to fill the op cache before looping begins and we make use of the faster op cache rate (8/9 ops/cy).

This patch proposes we instead cap the size of the LoopMicroOpBufferSize based off the maximum rate from the op cache (znver3 = 8op/cy, znver4 = 9op/cy) and the branch misprediction penalty from the opcache (~12cy) as a estimate of the useful number of ops we can unroll a loop by before mispredictions are likely to cause stalls. This isn't a perfect metric, but does try to be closer to the spirit of how we use LoopMicroOpBufferSize in the compiler vs the size of a similar naming buffer in the cpu.

2024-05-16 14:44:00 +01:00

AArch64

[AArch64] Improve cost model for legal subvec insert/extract (#81135 )

2024-03-04 16:17:01 +00:00

AMDGPU

[LoopUnroll] Consider simplified operands while retrieving TTI instruction cost (#70929 )

2024-02-06 17:01:38 +03:00

ARM

[Transforms] Convert tests to opaque pointers (NFC)

2024-02-05 11:57:34 +01:00

Hexagon

[NFC][Py Reformat] Reformat lit.local.cfg python files in llvm

2023-05-17 17:03:15 +02:00

PowerPC

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

RISCV

[LoopUnroll] Consider simplified operands while retrieving TTI instruction cost (#70929 )

2024-02-06 17:01:38 +03:00

WebAssembly

[InstCombine] Set disjoint flag when turning Add into Or. (#72702 )

2023-11-27 12:54:11 -08:00

X86

[X86] Reduce znver3/4 LoopMicroOpBufferSize to practical loop unrolling values (#91340 )

2024-05-16 14:44:00 +01:00

2004-05-13-DontUnrollTooMuch.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

2005-03-06-BadLoopInfoUpdate.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

2006-08-24-MultiBlockLoop.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

2007-04-16-PhiUpdate.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

2007-05-05-UnrollMiscomp.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

2007-05-09-UnknownTripCount.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

2007-11-05-Crash.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

2011-08-08-PhiUpdate.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

2011-08-09-IVSimplify.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

2011-08-09-PhiUpdate.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

2011-10-01-NoopTrunc.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

2012-04-09-unroll-indirectbr.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

add-folding-overflow.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

assume-timeout.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

basic.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

callbr.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

complete_unroll_profitability_with_assume.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

convergent.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

dce.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

debug-info.ll

[DebugInfo][RemoveDIs] Instrument inliner for non-instr debug-info (#72884 )

2023-11-26 21:24:29 +00:00

disable_nonforced_count.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

disable_nonforced_enable.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

disable_nonforced_full.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

disable_nonforced.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

disable-full-unroll-by-opt.ll

…

disable-loop-unrolling_forced.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

ephemeral.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

epilog_const_phi.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

followup.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-avoid-partial.ll

[LoopUnroll] Prevent LoopFullUnrollPass to perform partial/runtime unrolling

2023-04-13 10:21:24 +05:30

full-unroll-bad-cost.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

full-unroll-crashers.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-heuristics-2.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-heuristics-cmp.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-heuristics-dce.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-heuristics-geps.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-heuristics-phi-prop.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

full-unroll-heuristics.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

full-unroll-invariant.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

full-unroll-keep-first-exit.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

full-unroll-one-unpredictable-exit.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

FullUnroll.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

gh-issue77118-broken-lcssa-form.ll

[SimplifyIndVar] LCSSA form is destroyed by simplifyLoopIVs, preserve it (#78696 )

2024-02-21 17:51:56 +05:30

guard-cost-for-unrolling.ll

[TTI] Treat AND/OR with widenable conditions as free of cost

2023-03-14 20:55:17 +07:00

high-cost-trip-count-computation.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

ignore-annotation-intrinsic-cost.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

invalidate_right_loop.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

loop-block-disposition-after-full-unroll.ll

[LoopUnroll] Add cache verification failure test case.

2022-09-26 14:25:37 +01:00

loop-block-disposition-after-loop-peeling.ll

[LoopPeel] Clear dispositions after peeling

2023-07-19 10:39:59 +02:00

loop-branch-folding.ll

[LoopUnroll] Make use of MaxTripCount for loops with "#pragma unroll" (#74703 )

2023-12-08 19:43:10 +08:00

loop-remarks-with-hotness.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

loop-remarks.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

multiple-exits.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

noalias.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

nonlatchcondbr.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

not-rotated.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

opt-levels.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

optsize-loop-size.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

partial-unroll-const-bounds.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

partial-unroll-dead-instructions.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

partial-unroll-non-latch-exit.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

partially-unroll-unconditional-latch.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

peel-branch-weights.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop2.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-loop-and-unroll.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-loop-conditions-pgo-1.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop-conditions-pgo-2.ll

[NFC] Port all LoopUnroll tests to -passes= syntax

2022-12-08 02:38:47 +03:00

peel-loop-conditions.ll

[Transforms] Convert tests to opaque pointers (NFC)

2024-02-05 11:57:34 +01:00

peel-loop-inner.ll

Revert "reland e5581df60a35 [SimplifyCFG] accumulate bonus insts cost"

2022-10-25 12:15:39 -04:00

peel-loop-irreducible.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop-negative.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop-nests.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop-noalias-scope-decl.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-loop-not-forced.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop-pgo-deopt-idom-2.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-loop-pgo-deopt-idom.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-loop-pgo-deopt.ll

[LoopPeeling] Fix weights updating of peeled off branches (#70094 )

2023-10-31 14:02:42 +01:00

peel-loop-pgo.ll

[LoopPeeling] Fix weights updating of peeled off branches (#70094 )

2023-10-31 14:02:42 +01:00

peel-loop-phi-analysis.ll

Expand loop peeling phi computation to handle binary ops and casts

2022-12-05 12:10:53 -05:00

peel-loop-scev-invalidate.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

peel-loop.ll

[InstCombine] Canonicalize constant GEPs to i8 source element type (#68882 )

2024-01-24 15:25:29 +01:00

peel-multiple-unreachable-exits.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

peel-to-turn-invariant-accesses-dereferenceable.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr10813.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr11361.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

pr14167.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

pr18861.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr27157.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

pr28132.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr31718.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr33437.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

pr45939-peel-count-and-complete-unroll.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

pr77842.ll

[LoopUnroll] Introduce PragmaUnrollFullMaxIterations as a hard cap on how many iterations we try to unroll (#78648 )

2024-02-05 17:01:00 -08:00

rebuild_lcssa.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

revert-D110922.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

revisit.ll

[StandardInstrumentation] Annotate loops with the function name (#90756 )

2024-05-03 14:13:59 -04:00

runtime-epilog-debuginfo.ll

[DebugInfo][RemoveDIs] Support cloning and remapping DPValues (#72546 )

2023-11-24 15:17:32 +00:00

runtime-exit-phi-scev-invalidation.ll

[SCEVExpander] Attempt to reinfer flags dropped due to CSE (#72431 )

2023-12-07 13:20:36 -08:00

runtime-i128.ll

[LoopUnroll] Fix missing sign extension

2024-02-01 16:08:25 +01:00

runtime-li.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop1.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop2.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop3.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop4.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop5.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-loop-at-most-two-exits.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-loop-branchweight.ll

LoopUnrollRuntime: Add weights to all branches

2023-09-11 14:23:29 -07:00

runtime-loop-known-exit.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

runtime-loop-multiexit-dom-verify.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-loop-multiple-exits.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-loop-non-exiting-latch.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

runtime-loop.ll

LoopUnrollRuntime: Add weights to all branches

2023-09-11 14:23:29 -07:00

runtime-multiexit-heuristic.ll

[InstCombine] Improve foldICmpWithDominatingICmp with DomConditionCache (#75370 )

2023-12-14 21:02:10 +08:00

runtime-small-upperbound.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-unroll-assume-no-remainder.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

runtime-unroll-remainder.ll

[InstCombine] Set disjoint flag when turning Add into Or. (#72702 )

2023-11-27 12:54:11 -08:00

scev-dispositions-type-not-scevable.ll

[LoopUnroll] Add test for crash exposed by 9e931439.

2022-10-07 20:02:58 +01:00

scev-forget-exit-phis.ll

[LoopUnroll] Forget exit values when making changes.

2022-10-18 15:12:24 +01:00

scev-invalidation-lcssa.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

scevunroll.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

shifted-tripcount.ll

[LoopUnroll] Add CSE to remove redundant loads after unrolling. (#83860 )

2024-05-02 11:01:24 +01:00

tripcount-overflow.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

unloop.ll

[NFC] Port all (but one) LoopUnroll tests to -passes= syntax

2022-12-07 20:15:43 +03:00

unroll-after-peel.ll

Revert "Reland "[SimplifyCFG] FoldBranchToCommonDest(): deal with mismatched IV's in PHI's in common successor block""

2022-12-20 18:36:42 +03:00

unroll-cleanup.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-cleanuppad.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-cost-symbolic-execute.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-count.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-header-exiting-with-phis-multiple-exiting-blocks.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-header-exiting-with-phis.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

unroll-heuristics-pgo.ll

LoopUnrollRuntime: Add weights to all branches

2023-09-11 14:23:29 -07:00

unroll-loads-cse.ll

[LoopUnroll] Add CSE to remove redundant loads after unrolling. (#83860 )

2024-05-02 11:01:24 +01:00

unroll-loop-invalidation.ll

update_analyze_test_checks: support output from LAA (#67584 )

2023-10-31 14:33:53 +00:00

unroll-maxcount.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-opt-attribute.ll

[LoopUnroll] Fold add chains during unrolling

2023-07-05 09:54:28 +02:00

unroll-pragmas-disabled.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-pragmas.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

unroll-preserve-scev-lcssa.ll

…

unroll-remove-redundant-dbg.ll

[LoopUnroll] Remove redundant debug instructions after blocks have been merged (#91246 )

2024-05-13 09:42:04 -07:00

unroll-unconditional-latch.ll

[LoopUnroll] Convert some tests to opaque pointers (NFC)

2022-12-23 16:35:26 +01:00

update-loop-info-in-subloops.ll

[opt] Don't translate legacy -analysis flag to require<analysis>

2022-10-07 14:54:34 -07:00

verify-dominators.ll

[LoopUnroll] Add test case exposing crash with d0907ce7ed9f.

2023-01-20 16:08:25 +00:00

wrong_assert_in_peeling.ll

[test] Remove duplicate RUN lines in Transform tests

2022-12-08 11:47:16 +01:00