…ndling.
Returns an error *and* a benchmark rather than an error *or* a
benchmark. This allows users to have custom error handling while still
being able to inspect the benchmark.
Apart from this small API change, this is an NFC.
This is an alternative to #74211.
This patch introduces the subprocess executor mode. Currently, this new
mode doesn't do anything fancy, just executing the same code that the
inprocess executor would do, but within a subprocess. This sets up the
ability to add in many more memory-related features in the future.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D151021
Revert "[llvm-exegesis] Introduce Subprocess Executor Mode"
This reverts commit 5e9173c43a9b97c8614e36d6f754317f731e71e9.
This reverts commit 4d618b52f6e05e41d35f56653cb36bf7d4dc794e.
Reverting the PID commit as it is currently breaking MinGW builds and
the way I'm checking for the presence of pid_t needs to be fixed and I
need to do some testing. The subprocess executor mode patch is a
dependent patch so also needs to be reverted and also needs some work as
it is currently failing tests where libpfm is installed and the kernel
version is less than 5.6.
This patch introduces the subprocess executor mode. Currently, this new
mode doesn't do anything fancy, just executing the same code that the
inprocess executor would do, but within a subprocess. This sets up the
ability to add in many more memory-related features in the future.
This reverts commit 0d4ef4ff01addbb40b9122a00d6b2f23104cbb3b.
This was causing build failures on certain platforms when built with
-Werror due to unused variable warnings in addition to causing build
failures on Linux systems with older kernel versions as kernels prior to
v5.15 don't support sys_pidfd_getpid. Reverting as I need to setup a
system to properly test the rest of the patches in this series.
Also reverts 8c6668fa42dba59ddc286ba256d71c1b9c5228b8 which fixed the
first issue so that the patch can actually be reverted.
This patch introduces the subprocess executor mode. Currently, this new
mode doesn't do anything fancy, just executing the same code that the
inprocess executor would do, but within a subprocess. This sets up the
ability to add in many more memory-related features in the future.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D151021
In order to better support adding in new implementations of
FunctionExecutor, this patch makes some small changes so that it is
easier to add new ones in. FunctionExecutorImpl is renamed to
InProcessFunctionExecutorImpl to better reflect how it will be placed
relative to the soon-to-be introduced subprocess executor and a new
function is created to create executors so selection can be done more
easily. In addition, a new CLI flag, -execution-mode, which can be used
to select between the different executors.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D151019
This patch refactors some code out of FunctionExecutorImpl into the base
class that should be common across all implementations of
FunctionExecutor. Particularly, this patch factors out
accumulateCounterValues, and also factors out runAndSample, moving
implementation specific code into a new runWithCounter function. This
makes adding new implementations of FunctinExecutor easier.
Reviewed By: gchatelet
Differential Revision: https://reviews.llvm.org/D148079
This completes the FIXME listed in FunctionExecutor in regards to
deprecating this function. It simply makes the appropriate call into
runAndSample and grabs the first counter value. This patch completely
removes the function, moving that logic into the callers (currently only
uopsBenchmarkRunner). This makes creating new FunctionExecutors easier
as an implementation no longer needs to worry about this detail.
Reviewed By: gchatelet
Differential Revision: https://reviews.llvm.org/D147878
When llvm-exegesis was first introduced, it only supported benchmarking
individual instructions, hence the name for the data structure storing
the data corresponding to a benchmark being called InstructionBenchmark
made sense. However, now that benchmarking arbitrary snippets is
supported, InstructionBenchmark doesn't correspond to a single
instruction. This patch refactors InstructionBenchmark to be called
Benchmark to clean up this little bit of technical debt.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D146884
While "skip measurements mode" is super useful for test coverage,
i've come to discover it's trade-offs. It still calls back-end
to actually codegen the target assembly, and that is what is taking
80%+ of the time regardless of whether or not we skip the measurements.
On the other hand, just being able to see that exegesis can come up
with a snippet to measure something, is already very useful,
and takes maybe a second for a all-opcode sweep.
Reviewed By: gchatelet
Differential Revision: https://reviews.llvm.org/D140702
We can run as many `getRunnableConfiguration()` in parallel as we want,
but `runConfiguration()` must be run *completely* standalone from everything.
This is a step towards enabling threading.
If `BenchmarkRunner::runConfiguration()` deals with more than a single
repetitor, tasking will be less straight-forward to implement.
But i think dealing with that in it's callee is even more readable.
Sometimes we only want to ensure that we can produce snippets (all the way
through `SnippetRepetitor`!), but don't care for the execution.
E.g. all of our tests are this way.
I've built LLVM without PFM and removed my CPU from `X86PfmCounters.td`,
and this produces the expected results in that configuration.
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D139448
I really needed this, like, factually, yesterday,
when verifying dependency breaking idioms for AMD Zen 3 scheduler model.
Consider the following example:
```
$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=duplicate
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-4a7e50.o
---
mode: inverse_throughput
key:
instructions:
- 'VPXORYrr YMM0 YMM0 YMM0'
config: ''
register_initial_values: []
cpu_name: znver3
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 1000000
measurements:
- { key: inverse_throughput, value: 0.31025, per_snippet_value: 0.31025 }
error: ''
info: ''
assembled_snippet: C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C3
...
```
What does it tell us?
So wait, it can only execute ~3 x86 AVX YMM PXOR zero-idioms per cycle?
That doesn't seem right. That's even less than there are pipes supporting this type of op.
Now, second example:
```
$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=loop
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-2418b5.o
---
mode: inverse_throughput
key:
instructions:
- 'VPXORYrr YMM0 YMM0 YMM0'
config: ''
register_initial_values: []
cpu_name: znver3
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 1000000
measurements:
- { key: inverse_throughput, value: 1.00011, per_snippet_value: 1.00011 }
error: ''
info: ''
assembled_snippet: 49B80800000000000000C5FDEFC0C5FDEFC04983C0FF75F2C3
...
```
Now that's just worse. Due to the looping, the throughput completely plummeted,
and now we can only do a single instruction/cycle!?
That's not great.
And final example:
```
$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=loop --loop-body-size=1000
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-c402e2.o
---
mode: inverse_throughput
key:
instructions:
- 'VPXORYrr YMM0 YMM0 YMM0'
config: ''
register_initial_values: []
cpu_name: znver3
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 1000000
measurements:
- { key: inverse_throughput, value: 0.167087, per_snippet_value: 0.167087 }
error: ''
info: ''
assembled_snippet: 49B80800000000000000C5FDEFC0C5FDEFC04983C0FF75F2C3
...
```
So if we merge the previous two approaches, do duplicate this single-instruction snippet 1000x
(loop-body-size/instruction count in snippet), and run a loop with 1000 iterations
over that duplicated/unrolled snippet, the measured throughput goes through the roof,
up to 5.9 instructions/cycle, which finally tells us that this idiom is zero-cycle!
Reviewed By: courbet
Differential Revision: https://reviews.llvm.org/D102522
LBR contains (up to) 16 entries for last x branches and the X86LBRCounter (from D77422) should be able to return all those.
Currently, it just returns the latest entry, which could lead to mis-leading measurements.
This patch aslo changes the LatencyBenchmarkRunner to accommodate multi-value readings.
https://reviews.llvm.org/D81050
Summary:
As noted in documentation, different repetition modes have different trade-offs:
> .. option:: -repetition-mode=[duplicate|loop]
>
> Specify the repetition mode. `duplicate` will create a large, straight line
> basic block with `num-repetitions` copies of the snippet. `loop` will wrap
> the snippet in a loop which will be run `num-repetitions` times. The `loop`
> mode tends to better hide the effects of the CPU frontend on architectures
> that cache decoded instructions, but consumes a register for counting
> iterations.
Indeed. Example:
>>! In D74156#1873657, @lebedev.ri wrote:
> At least for `CMOV`, i'm seeing wildly different results
> | | Latency | RThroughput |
> | duplicate | 1 | 0.8 |
> | loop | 2 | 0.6 |
> where latency=1 seems correct, and i'd expect the througput to be close to 1/2 (since there are two execution units).
This isn't great for analysis, at least for schedule model development.
As discussed in excruciating detail in
>>! In D74156#1924514, @gchatelet wrote:
>>>! In D74156#1920632, @lebedev.ri wrote:
>> ... did that explanation of the question i'm having made any sense?
>
> Thx for digging in the conversation !
> Ok it makes more sense now.
>
> I discussed it a bit with @courbet:
> - We want the analysis tool to stay simple so we'd rather not make it knowledgeable of the repetition mode.
> - We'd like to still be able to select either repetition mode to dig into special cases
>
> So we could add a third `min` repetition mode that would run both and take the minimum. It could be the default option.
> Would you have some time to look what it would take to add this third mode?
there appears to be an agreement that it is indeed sub-par,
and that we should provide an optional, measurement (not analysis!) -time
way to rectify the situation.
However, the solutions isn't entirely straight-forward.
We can just add an actual 'multiplexer' `MinSnippetRepetitor`, because
if we just concatenate snippets produced by `DuplicateSnippetRepetitor`
and `LoopSnippetRepetitor` and run+measure that, the measurement will
naturally be different from what we'd get by running+measuring
them separately and taking the min.
([[ https://www.wolframalpha.com/input/?i=%28x%2By%29%2F2+%21%3D+min%28x%2C+y%29 | `time(D+L)/2 != min(time(D), time(L))` ]])
Also, it seems best to me to have a single snippet instead of generating
a snippet per repetition mode, since the only difference here is that the
loop repetition mode reserves one register for loop counter.
As far as i can tell, we can either teach `BenchmarkRunner::runConfiguration()`
to produce a single report given multiple repetitors (as in the patch),
or do that one layer higher - don't modify `BenchmarkRunner::runConfiguration()`,
produce multiple reports, don't actually print each one, but aggregate them somehow
and only print the final one.
Initially i've gone ahead with the latter approach, but it didn't look like a natural fit;
the former (as in the diff) does seem like a better fit to me.
There's also a question of the test coverage. It sure currently does work here:
```
$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=duplicate
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-8fb949.o
---
mode: inverse_throughput
key:
instructions:
- 'CMOV64rr RAX RAX R11 i_0x0'
- 'CMOV64rr RBP RBP R15 i_0x0'
- 'CMOV64rr RBX RBX RBX i_0x0'
- 'CMOV64rr RCX RCX RBX i_0x0'
- 'CMOV64rr RDI RDI R10 i_0x0'
- 'CMOV64rr RDX RDX RAX i_0x0'
- 'CMOV64rr RSI RSI RAX i_0x0'
- 'CMOV64rr R8 R8 R8 i_0x0'
- 'CMOV64rr R9 R9 RDX i_0x0'
- 'CMOV64rr R10 R10 RBX i_0x0'
- 'CMOV64rr R11 R11 R14 i_0x0'
- 'CMOV64rr R12 R12 R9 i_0x0'
- 'CMOV64rr R13 R13 R12 i_0x0'
- 'CMOV64rr R14 R14 R15 i_0x0'
- 'CMOV64rr R15 R15 R13 i_0x0'
config: ''
register_initial_values:
- 'RAX=0x0'
- 'R11=0x0'
- 'EFLAGS=0x0'
- 'RBP=0x0'
- 'R15=0x0'
- 'RBX=0x0'
- 'RCX=0x0'
- 'RDI=0x0'
- 'R10=0x0'
- 'RDX=0x0'
- 'RSI=0x0'
- 'R8=0x0'
- 'R9=0x0'
- 'R14=0x0'
- 'R12=0x0'
- 'R13=0x0'
cpu_name: bdver2
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
- { key: inverse_throughput, value: 0.819, per_snippet_value: 12.285 }
error: ''
info: instruction has tied variables, using static renaming.
assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BF000000000000000048BB000000000000000048B9000000000000000048BF000000000000000049BA000000000000000048BA000000000000000048BE000000000000000049B8000000000000000049B9000000000000000049BE000000000000000049BC000000000000000049BD0000000000000000490F40C3490F40EF480F40DB480F40CB490F40FA480F40D0480F40F04D0F40C04C0F40CA4C0F40D34D0F40DE4D0F40E14D0F40EC4D0F40F74D0F40FD490F40C35B415C415D415E415F5DC3
...
$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=loop
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-051eb3.o
---
mode: inverse_throughput
key:
instructions:
- 'CMOV64rr RAX RAX R11 i_0x0'
- 'CMOV64rr RBP RBP RSI i_0x0'
- 'CMOV64rr RBX RBX R9 i_0x0'
- 'CMOV64rr RCX RCX RSI i_0x0'
- 'CMOV64rr RDI RDI RBP i_0x0'
- 'CMOV64rr RDX RDX R9 i_0x0'
- 'CMOV64rr RSI RSI RDI i_0x0'
- 'CMOV64rr R9 R9 R12 i_0x0'
- 'CMOV64rr R10 R10 R11 i_0x0'
- 'CMOV64rr R11 R11 R9 i_0x0'
- 'CMOV64rr R12 R12 RBP i_0x0'
- 'CMOV64rr R13 R13 RSI i_0x0'
- 'CMOV64rr R14 R14 R14 i_0x0'
- 'CMOV64rr R15 R15 R10 i_0x0'
config: ''
register_initial_values:
- 'RAX=0x0'
- 'R11=0x0'
- 'EFLAGS=0x0'
- 'RBP=0x0'
- 'RSI=0x0'
- 'RBX=0x0'
- 'R9=0x0'
- 'RCX=0x0'
- 'RDI=0x0'
- 'RDX=0x0'
- 'R12=0x0'
- 'R10=0x0'
- 'R13=0x0'
- 'R14=0x0'
- 'R15=0x0'
cpu_name: bdver2
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
- { key: inverse_throughput, value: 0.6083, per_snippet_value: 8.5162 }
error: ''
info: instruction has tied variables, using static renaming.
assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000048BE000000000000000048BB000000000000000049B9000000000000000048B9000000000000000048BF000000000000000048BA000000000000000049BC000000000000000049BA000000000000000049BD000000000000000049BE000000000000000049BF000000000000000049B80200000000000000490F40C3480F40EE490F40D9480F40CE480F40FD490F40D1480F40F74D0F40CC4D0F40D34D0F40D94C0F40E54C0F40EE4D0F40F64D0F40FA4983C0FF75C25B415C415D415E415F5DC3
...
$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=min
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-c7a47d.o
Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-2581f1.o
---
mode: inverse_throughput
key:
instructions:
- 'CMOV64rr RAX RAX R11 i_0x0'
- 'CMOV64rr RBP RBP R10 i_0x0'
- 'CMOV64rr RBX RBX R10 i_0x0'
- 'CMOV64rr RCX RCX RDX i_0x0'
- 'CMOV64rr RDI RDI RAX i_0x0'
- 'CMOV64rr RDX RDX R9 i_0x0'
- 'CMOV64rr RSI RSI RAX i_0x0'
- 'CMOV64rr R9 R9 RBX i_0x0'
- 'CMOV64rr R10 R10 R12 i_0x0'
- 'CMOV64rr R11 R11 RDI i_0x0'
- 'CMOV64rr R12 R12 RDI i_0x0'
- 'CMOV64rr R13 R13 RDI i_0x0'
- 'CMOV64rr R14 R14 R9 i_0x0'
- 'CMOV64rr R15 R15 RBP i_0x0'
config: ''
register_initial_values:
- 'RAX=0x0'
- 'R11=0x0'
- 'EFLAGS=0x0'
- 'RBP=0x0'
- 'R10=0x0'
- 'RBX=0x0'
- 'RCX=0x0'
- 'RDX=0x0'
- 'RDI=0x0'
- 'R9=0x0'
- 'RSI=0x0'
- 'R12=0x0'
- 'R13=0x0'
- 'R14=0x0'
- 'R15=0x0'
cpu_name: bdver2
llvm_triple: x86_64-unknown-linux-gnu
num_repetitions: 10000
measurements:
- { key: inverse_throughput, value: 0.6073, per_snippet_value: 8.5022 }
error: ''
info: instruction has tied variables, using static renaming.
assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BA000000000000000048BB000000000000000048B9000000000000000048BA000000000000000048BF000000000000000049B9000000000000000048BE000000000000000049BC000000000000000049BD000000000000000049BE000000000000000049BF0000000000000000490F40C3490F40EA490F40DA480F40CA480F40F8490F40D1480F40F04C0F40CB4D0F40D44C0F40DF4C0F40E74C0F40EF4D0F40F14C0F40FD490F40C3490F40EA5B415C415D415E415F5DC35541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BA000000000000000048BB000000000000000048B9000000000000000048BA000000000000000048BF000000000000000049B9000000000000000048BE000000000000000049BC000000000000000049BD000000000000000049BE000000000000000049BF000000000000000049B80200000000000000490F40C3490F40EA490F40DA480F40CA480F40F8490F40D1480F40F04C0F40CB4D0F40D44C0F40DF4C0F40E74C0F40EF4D0F40F14C0F40FD4983C0FF75C25B415C415D415E415F5DC3
...
```
but i open to suggestions as to how test that.
I also have gone with the suggestion to default to this new mode.
This was irking me for some time, so i'm happy to finally see progress here.
Looking forward to feedback.
Reviewers: courbet, gchatelet
Reviewed By: courbet, gchatelet
Subscribers: mstojanovic, RKSimon, llvm-commits, courbet, gchatelet
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D76921
Followup to D74085.
Replace the use of `report_fatal_error()` with returning the error to
`llvm-exegesis.cpp` and handling it there.
To facilitate this, a new `Error` type has been added which is only used
to log errors to the yaml output.
Differential Revision: https://reviews.llvm.org/D74215
Summary:
Before this change the Executable function was made by duplicating the
snippet. This change adds a --repetion-mode={loop|duplicate} flag that
allows choosing between this behaviour and wrapping the snippet instructions
in a loop.
The new mode can help measurements when the snippet fits in the DSB by
short-cirtcuiting decoding. The loop adds a dec + jmp to the measurements, but
since these are not part of the critical path, they execute in parallel
with the measured code and do not impact measurements in practice.
Overview of the change:
- New SnippetRepetitor abstraction that handles repeating the snippet.
The assembler delegates repeating the instructions to this class.
- ExegesisTarget learns how to decrement loop counter and jump.
- Some refactoring of the assembler into FunctionFiller/BasicBlockFiller.
Reviewers: gchatelet
Subscribers: mgorny, tschuett, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D68125
llvm-svn: 373083
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
llvm-svn: 369013
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
This allows simplifying references of llvm::foo with foo when the needs
come in the future.
Reviewers: courbet, gchatelet
Reviewed By: gchatelet
Subscribers: javed.absar, tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D53455
llvm-svn: 344922
Summary:
We try to recover gracefully on instructions that would crash the
program.
This includes some refactoring of runMeasurement() implementations.
Reviewers: gchatelet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D53371
llvm-svn: 344695
Summary:
The snippet-generation part goes to the SnippetGenerator class.
This will allow benchmarking arbitrary code (see PR38437).
Reviewers: gchatelet
Subscribers: mgorny, tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D51979
llvm-svn: 342117
Summary:
This enables the X86-specific X86FloatingPointStackifierPass, and allow
llvm-exegesis to generate and measure X87 latency/uops for some FP ops.
Reviewers: gchatelet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48592
llvm-svn: 335815
Summary:
This allows targets to override code generation for some instructions.
As an example of override, this also moves ad-hoc instruction filtering
for X86 into the X86 ExegesisTarget.
Reviewers: gchatelet
Subscribers: mgorny, tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48587
llvm-svn: 335582
Summary:
This ensures that the snippet always sees the same values for registers,
making measurements reproducible.
This will also allow exploring different values.
Reviewers: gchatelet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48542
llvm-svn: 335465
Summary: Introducing a Prototype object to capture Variables that must be set but keeps degrees of freedom as Invalid. This allows exploring non constraint variables later on.
Reviewers: courbet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48316
llvm-svn: 335105
Summary:
On hover, the whole asm snippet is displayed, including operands.
This requires the actual assembly output instead of just the MCInsts:
This is because some pseudo-instructions get lowered to actual target
instructions during codegen (e.g. ABS_Fp32 -> SSE or X87).
Reviewers: gchatelet
Subscribers: mgorny, tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48164
llvm-svn: 334805
Summary: Previous design was relying on the 'mutate' keyword and was quite confusing. This version separate mutable from immutable data and makes it clearer what changes and what doesn't.
Reviewers: courbet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D48020
llvm-svn: 334596
Summary: BenchmarkRunner subclasses can now create many configurations - although this patch still generates one.
Reviewers: courbet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D47877
llvm-svn: 334197
Summary: This is the first step to have the BenchmarkRunner create and measure many different configurations (different initial values for instance).
Reviewers: courbet
Subscribers: tschuett, llvm-commits
Differential Revision: https://reviews.llvm.org/D47826
llvm-svn: 334169
