Parameterization

---

Sample: IntroParams

You can mark one or several fields or properties in your class by the [Params] attribute. In this attribute, you can specify set of values. Every value must be a compile-time constant. As a result, you will get results for each combination of params values.

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroParams
    {
        [Params(100, 200)]
        public int A { get; set; }

        [Params(10, 20)]
        public int B { get; set; }

        [Benchmark]
        public void Benchmark() => Thread.Sleep(A + B + 5);
    }
}

Output

   Method  |      Median |    StdDev |   A |  B
---------- |------------ |---------- |---- |---
 Benchmark | 115.3325 ms | 0.0242 ms | 100 | 10
 Benchmark | 125.3282 ms | 0.0245 ms | 100 | 20
 Benchmark | 215.3024 ms | 0.0375 ms | 200 | 10
 Benchmark | 225.2710 ms | 0.0434 ms | 200 | 20

Links

• Parameterization
• The permanent link to this sample: Sample: IntroParams

---

Sample: IntroParamsSource

In case you want to use a lot of values, you should use [ParamsSource] You can mark one or several fields or properties in your class by the [Params] attribute. In this attribute, you have to specify the name of public method/property which is going to provide the values (something that implements IEnumerable). The source must be within benchmarked type!

Source code

using System.Collections.Generic;
using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroParamsSource
    {
        // property with public setter
        [ParamsSource(nameof(ValuesForA))]
        public int A { get; set; }

        // public field
        [ParamsSource(nameof(ValuesForB))]
        public int B;

        // public property
        public IEnumerable<int> ValuesForA => new[] { 100, 200 };

        // public static method
        public static IEnumerable<int> ValuesForB() => new[] { 10, 20 };

        [Benchmark]
        public void Benchmark() => Thread.Sleep(A + B + 5);
    }
}

Output

   Method  |      Median |    StdDev |   A |  B
---------- |------------ |---------- |---- |---
 Benchmark | 115.3325 ms | 0.0242 ms | 100 | 10
 Benchmark | 125.3282 ms | 0.0245 ms | 100 | 20
 Benchmark | 215.3024 ms | 0.0375 ms | 200 | 10
 Benchmark | 225.2710 ms | 0.0434 ms | 200 | 20

Remarks

A remark about IParam.

You don't need to use IParam anymore since 0.11.0. Just use complex types as you wish and override ToString method to change the display names used in the results.

Links

• Parameterization
• The permanent link to this sample: Sample: IntroParamsSource

---

Sample: IntroParamsAllValues

If you want to use all possible values of an enum or another type with a small number of values, you can use the [ParamsAllValues] attribute, instead of listing all the values by hand. The types supported by the attribute are:

• bool
• any enum that is not marked with [Flags]
• Nullable<T>, where T is an enum or boolean

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    [DryJob]
    public class IntroParamsAllValues
    {
        public enum CustomEnum
        {
            A,
            BB,
            CCC
        }

        [ParamsAllValues]
        public CustomEnum E { get; set; }

        [ParamsAllValues]
        public bool? B { get; set; }

        [Benchmark]
        public void Benchmark()
        {
            Thread.Sleep(
                E.ToString().Length * 100 +
                (B == true ? 20 : B == false ? 10 : 0));
        }
    }
}

Output

    Method |   E |     B |     Mean | Error |
---------- |---- |------ |---------:|------:|
 Benchmark |   A |     ? | 101.9 ms |    NA |
 Benchmark |   A | False | 111.9 ms |    NA |
 Benchmark |   A |  True | 122.3 ms |    NA |
 Benchmark |  BB |     ? | 201.5 ms |    NA |
 Benchmark |  BB | False | 211.8 ms |    NA |
 Benchmark |  BB |  True | 221.4 ms |    NA |
 Benchmark | CCC |     ? | 301.8 ms |    NA |
 Benchmark | CCC | False | 312.3 ms |    NA |
 Benchmark | CCC |  True | 322.2 ms |    NA |

// * Legends *
  E     : Value of the 'E' parameter
  B     : Value of the 'B' parameter

Links

• Parameterization
• The permanent link to this sample: Sample: IntroParamsAllValues

---

Sample: IntroArguments

As an alternative to using [Params], you can specify arguments for your benchmarks. There are several ways to do it (described below).

Important

InProcessToolchain does not support Arguments (yet!). See #687 for more details.

The [Arguments] allows you to provide a set of values. Every value must be a compile-time constant (it's C# lanugage limitation for attributes in general). You can also combine [Arguments] with [Params]. As a result, you will get results for each combination of params values.

Source code

using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArguments
    {
        [Params(true, false)] // Arguments can be combined with Params
        public bool AddExtra5Milliseconds;

        [Benchmark]
        [Arguments(100, 10)]
        [Arguments(100, 20)]
        [Arguments(200, 10)]
        [Arguments(200, 20)]
        public void Benchmark(int a, int b)
        {
            if (AddExtra5Milliseconds)
                Thread.Sleep(a + b + 5);
            else
                Thread.Sleep(a + b);
        }
    }
}

Output

|    Method | AddExtra5Miliseconds |   a |  b |     Mean |     Error |    StdDev |
|---------- |--------------------- |---- |--- |---------:|----------:|----------:|
| Benchmark |                False | 100 | 10 | 110.1 ms | 0.0056 ms | 0.0044 ms |
| Benchmark |                False | 100 | 20 | 120.1 ms | 0.0155 ms | 0.0138 ms |
| Benchmark |                False | 200 | 10 | 210.2 ms | 0.0187 ms | 0.0175 ms |
| Benchmark |                False | 200 | 20 | 220.3 ms | 0.1055 ms | 0.0986 ms |
| Benchmark |                 True | 100 | 10 | 115.3 ms | 0.1375 ms | 0.1286 ms |
| Benchmark |                 True | 100 | 20 | 125.3 ms | 0.1212 ms | 0.1134 ms |
| Benchmark |                 True | 200 | 10 | 215.4 ms | 0.0779 ms | 0.0691 ms |
| Benchmark |                 True | 200 | 20 | 225.4 ms | 0.0775 ms | 0.0725 ms | 

Links

• Parameterization
• The permanent link to this sample: Sample: IntroArguments

---

Sample: IntroArgumentsSource

In case you want to use a lot of values, you should use [ArgumentsSource].

You can mark one or several fields or properties in your class by the [ArgumentsSource] attribute. In this attribute, you have to specify the name of public method/property which is going to provide the values (something that implements IEnumerable). The source must be within benchmarked type!

Source code

using System;
using System.Collections.Generic;
using System.Threading;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArgumentsSource
    {
        [Benchmark]
        [ArgumentsSource(nameof(Numbers))]
        public double ManyArguments(double x, double y) => Math.Pow(x, y);

        public IEnumerable<object[]> Numbers() // for multiple arguments it's an IEnumerable of array of objects (object[])
        {
            yield return new object[] { 1.0, 1.0 };
            yield return new object[] { 2.0, 2.0 };
            yield return new object[] { 4.0, 4.0 };
            yield return new object[] { 10.0, 10.0 };
        }

        [Benchmark]
        [ArgumentsSource(nameof(TimeSpans))]
        public void SingleArgument(TimeSpan time) => Thread.Sleep(time);

        public IEnumerable<object> TimeSpans() // for single argument it's an IEnumerable of objects (object)
        {
            yield return TimeSpan.FromMilliseconds(10);
            yield return TimeSpan.FromMilliseconds(100);
        }
    }
}

Output

| Method |  x |  y |      Mean |     Error |    StdDev |
|------- |--- |--- |----------:|----------:|----------:|
|    Pow |  1 |  1 |  9.360 ns | 0.0190 ns | 0.0149 ns |
|    Pow |  2 |  2 | 40.624 ns | 0.3413 ns | 0.3192 ns |
|    Pow |  4 |  4 | 40.537 ns | 0.0560 ns | 0.0524 ns |
|    Pow | 10 | 10 | 40.395 ns | 0.3274 ns | 0.3063 ns |

Another example

If the values are complex types you need to override ToString method to change the display names used in the results.

[DryJob]
public class WithNonPrimitiveArgumentsSource
{
    [Benchmark]
    [ArgumentsSource(nameof(NonPrimitive))]
    public void Simple(SomeClass someClass, SomeStruct someStruct)
    {
        for (int i = 0; i < someStruct.RangeEnd; i++)
            Console.WriteLine($"// array.Values[{i}] = {someClass.Values[i]}");
    }

    public IEnumerable<object[]> NonPrimitive()
    {
        yield return new object[] { new SomeClass(Enumerable.Range(0, 10).ToArray()), new SomeStruct(10) };
        yield return new object[] { new SomeClass(Enumerable.Range(0, 15).ToArray()), new SomeStruct(15) };
    }

    public class SomeClass
    {
        public SomeClass(int[] initialValues) => Values = initialValues.Select(val => val * 2).ToArray();

        public int[] Values { get; }

        public override string ToString() => $"{Values.Length} items";
    }

    public struct SomeStruct
    {
        public SomeStruct(int rangeEnd) => RangeEnd = rangeEnd;

        public int RangeEnd { get; }

        public override string ToString() => $"{RangeEnd}";
    }
}

| Method | someClass | someStruct |     Mean | Error |
|------- |---------- |----------- |---------:|------:|
| Simple |  10 items |         10 | 887.2 us |    NA |
| Simple |  15 items |         15 | 963.1 us |    NA |

Links

• Parameterization
• The permanent link to this sample: Sample: IntroArgumentsSource

---

Sample: IntroArrayParam

Warning

The cost of creating the arguments is not included in the benchmark.

So if you want to pass an array as an argument, we are going to allocate it before running the benchmark, and the benchmark will not include this operation.

Source code

using System;
using System.Collections.Generic;
using System.Linq;
using BenchmarkDotNet.Attributes;

namespace BenchmarkDotNet.Samples
{
    public class IntroArrayParam
    {
        [Benchmark]
        [ArgumentsSource(nameof(Data))]
        public int ArrayIndexOf(int[] array, int value)
            => Array.IndexOf(array, value);

        [Benchmark]
        [ArgumentsSource(nameof(Data))]
        public int ManualIndexOf(int[] array, int value)
        {
            for (int i = 0; i < array.Length; i++)
                if (array[i] == value)
                    return i;

            return -1;
        }

        public IEnumerable<object[]> Data()
        {
            yield return new object[] { new int[] { 1, 2, 3 }, 4 };
            yield return new object[] { Enumerable.Range(0, 100).ToArray(), 4 };
            yield return new object[] { Enumerable.Range(0, 100).ToArray(), 101 };
        }
    }
}

Output

|        Method |      array | value |      Mean |     Error |    StdDev | Allocated |
|-------------- |----------- |------ |----------:|----------:|----------:|----------:|
|  ArrayIndexOf | Array[100] |     4 | 15.558 ns | 0.0638 ns | 0.0597 ns |       0 B |
| ManualIndexOf | Array[100] |     4 |  5.345 ns | 0.0668 ns | 0.0625 ns |       0 B |
|  ArrayIndexOf |   Array[3] |     4 | 14.334 ns | 0.1758 ns | 0.1558 ns |       0 B |
| ManualIndexOf |   Array[3] |     4 |  2.758 ns | 0.0905 ns | 0.1208 ns |       0 B |
|  ArrayIndexOf | Array[100] |   101 | 78.359 ns | 1.8853 ns | 2.0955 ns |       0 B |
| ManualIndexOf | Array[100] |   101 | 80.421 ns | 0.6391 ns | 0.5978 ns |       0 B |

Links

• Parameterization
• The permanent link to this sample: Sample: IntroArrayParam

---