Benchmark Programs in C#

Problem. You are wondering if the C# programming language is fast. You question whether it is worth benchmarking programming languages. What else can you learn from careful benchmarks? Solution. Here’s a benchmarking overview using the C# programming language that touches on some important things about benchmarking.

::: Benchmarking tip :::

    Use the programs here to perform micro-benchmarks.
    Contextualize these results in the scale of other operations.
    Don't spend 99% of your time optimizing code that runs 1% of the time.

1. Benchmarking C#

Here we look at the kinds of loops that Dot Net Perls benchmarks are usually done with. This simple framework is what the author does his experiments with. He currently uses Vista which has much more accurate Stopwatch measurements in my experience. You will want to change m depending on the code of each iteration; it is best to start smaller and push the limit up.

=== Benchmarking Console application, 2 loops (C#) ===
    Use the two loops here to time code.
    The two stopwatches are set up to print out the time.
    Run in a new C# console application.
    Adjust m higher or lower based on how slow your iterations are.
    Always run in Release mode, never in VS, click the .exe yourself.
    Change the order of the tests.

using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;

class Program
{
    const int _max = 1000000;
    static void Main()
    {
        var s1 = Stopwatch.StartNew();
        for (int i = 0; i < _max; i++)
        {
        }
        s1.Stop();
        var s2 = Stopwatch.StartNew();
        for (int i = 0; i < _max; i++)
        {
        }
        s2.Stop();
        Console.WriteLine(((double)(s1.Elapsed.TotalMilliseconds * 1000 * 1000) / _max).ToString("0.00") + " ns");
        Console.WriteLine(((double)(s2.Elapsed.TotalMilliseconds * 1000 * 1000) / _max).ToString("0.00") + " ns");
        Console.Read();
    }

    static int _temp1;
    static int _temp2;

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Method1()
    {
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Method2()
    {
    }
}

=== Benchmarking Console application, 3 loops (C#) ===
    This prints out milliseconds.

using System;
using System.Diagnostics;

class Program
{
    static void Main()
    {
        const int m = 1000000;
        Stopwatch s1 = Stopwatch.StartNew();
        for (int i = 0; i < m; i++)
        {
        }
        s1.Stop();
        Stopwatch s2 = Stopwatch.StartNew();
        for (int i = 0; i < m; i++)
        {
        }
        s2.Stop();
        Stopwatch s3 = Stopwatch.StartNew();
        for (int i = 0; i < m; i++)
        {
        }
        s3.Stop();
        Console.WriteLine("{0},{1},{2}",
            s1.ElapsedMilliseconds,
            s2.ElapsedMilliseconds,
            s3.ElapsedMilliseconds);
        Console.Read();
    }
}

=== Benchmarking aspx file (ASP.NET and C#) ===

using System;
using System.Diagnostics;
using System.Web.UI;

public partial class _Default : Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        const int m = 1000000;
        Stopwatch s1 = Stopwatch.StartNew();
        for (int i = 0; i < m; i++)
        {
        }
        s1.Stop();
        Stopwatch s2 = Stopwatch.StartNew();
        for (int i = 0; i < m; i++)
        {
        }
        s2.Stop();
        Response.Write("Loop 1: ");
        Response.Write(s1.ElapsedMilliseconds);
        Response.Write("<br/>" +
            "Loop 2: ");
        Response.Write(s2.ElapsedMilliseconds);
    }
}

Notes on benchmarks. The benchmarking code has some problems, such as the second block rarely taking less time to execute due to unknown causes. Repeat and swap two loops if you are doubtful. The first program shown converts the results to nanoseconds, which are easier to understand and provide a way to compare different operations together easier. It is usually best to report results in nanoseconds or microseconds with the number of iterations used to divide the result. [C# Convert Nanoseconds, Microseconds and Milliseconds – dotnetperls.com]

2. Importance

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.” My interpretation: 97% of the time, don’t use optimizations on code that is not run often and that make it harder to understand and maintain.

Is benchmarking important? Yes. However, it is often not important in a practical sense for your current project. It encourages you to examine your code and find out what it is really doing.

Example optimization. The Dictionary collection in the BCL is a huge optimization and reduces lookup time to a constant O(1). However, developers sometimes write code that results in twice as many lookups. What did I learn from this? [C# Dictionary Lookup Comparison – dotnetperls.com]

How to dig deeper. The JIT optimizer in .NET makes code really fast, but it can’t fix everything. By benchmarking, I am able to tell what code is optimized. I gain insight into JITting.

3. Knowledge

With the .NET Reflector, we can see the internals of our C# code. However, the best way to tell that code is doing less on your Pentium is to time it. As computer scientists, we need to understand every part of how computer languages work. With benchmarking, I improved my knowledge of pipelining and locality of reference.

Improvement in speed
    Relative importance: Low (usually)

Improvement in knowledge
    Relative importance: Very high (sometimes)

4. Locality of reference

Locality of reference is important to benchmark. This gives us insight into how CPUs work, and what is essentially the core of computer science. Things that are near are faster to access. Astronomers will tell you the speed of light. Fundamental to science is locality of reference, and this carries over into the C# language.

5. Natural languages

Here we look at optimization from another point of view. As I have progressed as a writer, I have worked on making my English clearer and shorter to read. I think of this as optimization in the same way as with the C# language.

=== Slow English ===
    What I want to show to you next is this code, which adds
    one to a variable in each iteration of the loop.

=== Fast English
    The loop increments the variable.
    (Which would you rather read?)

6. Summary

Here we looked at example code for benchmarking the C# programming language. Benchmarking encourages careful thinking about your code. It saves nanoseconds from your software but greatly improves the depth of your understanding. With benchmarking, the author is able to grasp pipelining, file system caches, and locality of reference.

From here

Comments are closed.

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: