How to reduce complexity and know enough.

Quality

We have a great QA.

His code is like a Rolex.

You look at his gherkin, and smile.  It says what it's testing, in plain English, and there's nothing else on the page to confuse or distract you... just English.

"Why then, did our unit tests give me a feeling of dread?"  I asked myself.  It was quite obvious, that even though we used a neat, little NuGet package, called BDDfy to allow us to write English gherkins, our code was littered with private methods and other necessary evils:

private readonly Subject _subject;

MyTestClass()
{
_subject = new Subject();
}

[Fact]
public void WhenNoOneReadsYourBlog_WriteABunchOfCrap()
{
this
.Given(_ => _.NoOneReadsYourBlog())
.When(_ => _.YouFeelLikeWriting())
.Then(_ => _.JustWriteABunchOfCrap())
.BDDfy()
}

private async Task IAmADumbAnnoyingPrivateMethodThatNoOneWantsToSeeAsync()
{
var IMyInterface mockableMockObj = Substitute.For<IMyInterface>();
mockableMockObj.SomeStupidMethodThatNoOneReadingTheTestCaresAbout
.ShouldReturnSomethingElseInsteadOfNotDoingSomethingElse();

for(var i in someStupidComplicatedIteratableThing)
{
await WhyAmIEvenHereAsync();
}
}

"If we could just hide all the non-gherkin stuff, that would make the code more pleasing to read, understand, and work with," I thought to myself.  So I suggested to my team that we use a partial class.  We put the public methods, with the English definition of the tests in one file, and everything else in another.  The team loved the idea, so that is what we do.

Now, we still have many tests in a file (See how I organize unit tests here).  So, the question is, are the tests still useful, to understand what the system does?

Let me put it another way... how do you know enough?

Simplification

The company I work for, IHS Markit, is merging with S&P Global.  The combined company will have a hundred gazillion staff members, but only 6 divisions.  How do you know what S&P do?  You list the 6 divisions.

The universe consists of 200,000,000,000,000,000,000,000 stars.  Perhaps it can be summarized as a bag of galaxies.

It takes many years to learn about the human body, but you can summarize it as five things: head, neck, body, a pair of arms and a pair of legs.

Simplifying complexity into a tree, where each node has about 5 children, makes everything easy to understand.

Microservices is the modern, cool way to develop distributed systems.  AWS S3 consists of 300 services.  I haven't seen this architecture, but I imagine if they're just a list of services that call each other higgledy piggledy, then AWS has a big problem.  Our software has 30 services, and even that is a lot to try to understand.

My point is, that I think pretty much everything in software, and perhaps even life, needs to be broken down into a tree, limited to about 5 child nodes per parent to make it easy to understand.  Now, don't get me wrong... 5 is more of a thumb suck than anything else, but 5 is a handful.

You may be wondering, well, what if it can't be broken down?

• Maybe I have a property for each country... well, that's just a bag, collection, list, or array.
• Maybe my system just has 10 things it needs to do... categorize them... facilities, business logic, etc.

My hypothesis is that any group of items can be broken down into smaller groups... it just takes a bit of time to think up suitable categories, and re-organize things.

Code Organization

Problems with flat, rather than code organized in hierarchies, adding to complexity are:

• Private methods, private fields and private classes clutter the code, making it more difficult to understand than it needs to be.
• Long methods.

Normally we wouldn't organize classes in a flat structure.  We'd use a hierarchy of folders.  I think it would be interesting if a class or namespace was a folder.  Perhaps an IDE plugin could be written.  It might look like this:

MyApp.csproj

    Program.cs

    Utilities.cs

        DataTypes.cs

            Strings.cs

            Dates.cs

                Conversion.cs

                Validation.cs

        Files.cs

            Encryption.cs

            Persistence.cs

        

Contemplation

Consider code within classes too.

Would limiting methods to 5 per class or interface help to make it easier to understand?  The interface segregation principle suggests keeping interfaces "small" to prevent having classes that have to implement methods unnecessarily.  It may require a little less thought, to have alarm bells going off in your head when you reach 5 methods, to consider whether another interface is required.

Can limiting properties to 5 per class make them easier to work with?  I've seen some pretty large view models.  Code becomes much cleaner when they're broken down into a tree structure.  It can be tricky to change it later, so best to keep them small from the start. 

Can limiting lines to 5 per method make the logic easier to follow?  Uncle Bob says, "extract, extract, extract."  If you follow that, your methods will be pretty small.  Using the number 5 might be too artificial, but when you have to scroll to see your whole method, you've probably gone way too far.

I've heard a suggesting that methods should only have 3 parameters.  The suggestion is that if there are more, they should be in a class.

Disclaimer:  As I've only thought of  some of these ideas recently, I haven't experimented with these ideas very much.  So, I'm going to try them out, and see how it goes.

Thoughts?

Final Thoughts

A tree with a single root, and 5 children per parent, and a depth of 11 nodes, would have just under 10 million leaf nodes.  That's a hell of a lot of complexity, but simple enough to navigate.

There's no need to learn everything.  That's impossible.  I think the most important things to learn, as a programmer, are the capabilities of tools.  That way, whatever you need to build, you can pick the right tool, and then you just need to learn how to use it.  Navigate the tree of knowledge to the depth you need at the time.

Image credits:

https://www.freeimages.com/photo/skeleton-pocket-watch-back-1637098

https://www.freeimages.com/photo/nebula-space-astro-photo-astronomy-sky-1420873

https://www.freeimages.com/photo/burning-tree-1377053

https://www.freeimages.com/photo/dave-in-window-1553933

AWS Lambda vs Fargate

My manager, 7 finger Lucy

The following story is a re-enactment of what actually happened.  Names, dates, species and quite a lot more have been changed to protect those involved in the crimes against code...

Smoke emerged from management ears.  It had been months.  Six, perhaps, since our principal dev-ops ninja had started converting our twenty-eight Windows web services to run in containers.  Not a single container had been demoed yet.  "Stephen, we'd like you to take over this project, and... err... get it done quickly."

"Challenge accepted," I replied, retrieving an extra keyboard, pulling up my sleeves, and tightening my head band.  Browser open, my fingers sprang into action, "Google, what is...?"

I know what a container is, but err... yeah, my experience lacked... experience.

"So, principal dev..."  (I'll call her Bob)

"...Bob, what have we done so far?"

"600 hundred gazillion lines of pure code, she winked... YAML, Kubernetes, Terraform, nginX, Helm Charts, Docker, WSL 2, Artifactory and Windows Terminal.  She licked her lips.  Here, let me show you..."

Pete

The challenge, it appeared, seemed a bit more challenging that I had anticipated.  "Pete," I called our AWS consultant, because his name was Pete.

"Fargate," Pete replied.  A gong resounded.  I don't know why.  "Less problems, with Fargate, you will have."

I liked the idea.  I liked AWS.  Fargate was AWS.

I googled, "YAML, Kubernetes, Terraform, nginX, Helm Charts, Docker, WSL 2, Artifactory, Windows Terminal, Fargate."

Then I YouTubed, "YAML, Kubernetes, Terraform, nginX, Helm Charts, Docker, WSL 2, Artifactory, Windows Terminal, Fargate."

"I no longer have any more to teach you, my son," said the voice in my head.  It sounded like Pete.  "A container, in docker, you must create.  To Fargate, it must go."

"Kubernetes vs. Fargate," I typed into YouTube.  "EKS vs. ECS," it suggested.  "Fine, EKS vs ECS," I replied.  "ECS is simpler," it replied.

"Simple, I like simple," I told YouTube.

I ignored Bob's Git branch.  "600 hundred gazillion lines of pure code..." ignored.

I powered up PowerShell, installed WSL 2, formed my CloudFormation, setup a GitLab runner, bashed a new Bash script, docked a container in Docker, shipped it to ECR and Artifactory, and sailed it on to Fargate.

To be fair, it took me two months to release our first container.  There were hurdles, every step of the way, from setting up the GitLab runner to getting the containers to scale fast enough.  What I had achieved, however, was a custom solution that allowed us to 

• continue to deploy to our individual, development EC2 environments
• continue to be able to debug and step through any part or all of our application locally
• build and run auto-scaling Linux containers in multiple environments

The cherry on the cake, however, was that I could show a colleague what to do for the rest of the application, and by simply copying and pasting a Dockerfile (renaming the entry point), and configuring a few values in a JSON file (CPU, memory, storage, scaling profile) he was able to convert a web service.  Setting up the remaining 27 services took a couple of days.

I patted myself on the back.

"Lambda, an alternative to containers, is.  To Lambda, your code could go."

"But you said,..."

"Mmmm...  Simpler.  Simpler your solution could be."

I remembered Lambdas.  Python scripts, that timed out and ran out of memory... very quickly.  And we were charged... for every call!

"Que?"  I asked the voice in my head.

"Not just Python... all languages.  Compiled code and NuGet packages run, it can!"

"Quicker, your API calls must be."

"Less memory, your APIs use, they must."

"For every million calls, charged, you will be."

And so it was, that I found myself questioning whether or not I had made the right choice.  I think the gist is, that there are always going to be alternatives.  Often there isn't a right choice.  While prepping and doing research to find a better solution before starting is optimal, when deadlines crop up, there  may not be enough time to do enough researching and prototyping.  Sometimes it's necessary to ask the experts what they think, and go with that.  

In the case of Lambdas vs ECS Fargate, I think the most important considerations are firstly, can it actually be done as a Lambda?  Lambdas are, at the time of writing this, restricted to 15 minutes and 10GB of RAM.  Secondly, it's price.  If the code base is small, loads quickly and only needs to run for a few seconds at a time, Lambda could be cheaper.  For long running code, with large binaries, that doesn't need to scale up and down as quickly, Fargate is cheaper.  If you want to keep data cached in memory for a long time, Lambda is not going to work.  If you want to prototype something for free, and don't expect much traffic, Lambda might be the way to go.  As with many services in AWS, one has to do the price math, to make good decisions.

Here's a useful breakdown for anyone who would like to know more about the costs.

Picking a Better Data Store

I like Entity Framework.  It's like arriving at Hogwarts and trying out your wand for the first time .  Sometimes you wave it, and it transforms your rat into a perfect water goblet.  Other times it just blows up in your face, and you have to spend the rest of your high school years trying to master the correct incantation.

The question is, "Why do you have to have magic to transform your code into SQL?"  I used to say for unit testing, but if I want to test my units (whatever a unit is), I could just run them against an in-memory database.  Why do we need to transform a rat into a water goblet, when we could just go out and buy a water goblet, and keep our pet rat?  A rat is not a water goblet, neither is my Linq code TSQL, nor is my aggregate a table.

I use Entity Framework for two reasons: because it's fun, and to show off.  You could say that it makes the code more interesting.  It reminds me of AutoMapper... a library that automatically copies data from one object to another, if the property names match.  If they don't... you can extend your mapping to cater for it.

This is boring:

A.X = B.X;

A.Y = B.Z;

This is cool:

AutoMapperCadabra(A, B);

// TADA!

However, one is easy to debug, and one is not.  That could be a bumper sticker, "Magic... it's not easy to debug!"

Anyway... what was the point of this blog again?  Ah yes... picking a better data store.  What if there was no need for magic?  What if you could just put your aggregate in a database?

And that's why we use document databases.  But there's more to it:

Document databases are typically designed to scale horizontally, splitting the data up in shards on separate nodes.  A relational database typically requires either vertical scaling, or to replicate the entire database to multiple read nodes.

In a document database your aggregate is read and written sequentially... everything for your aggregate is in the same place.  It's not necessary to find bits and pieces all over the hard drive and bundle them together.

But... relational databases have their place.  You can index almost any piece of data, find it, and discover everything it's related to.  Want to find out something complex, like all the brands of wands used to create water goblets where the rat tail was still protruding from the water goblet?  You can!  Not so quick with a document database, unless you read everything.

They also have a schema, meaning that the format of every row is consistent.  This could be important, knowing that whenever you load a record, it it will be in the expected structure.

So, it depends what you need.  You could always dump your changes into a couple of queues and have multiple consumers build up different kinds of data stores, if you really want to.

SOLID

Nerdy Monk with Kittens

SOLID, YAGNI and DRY are acronyms commonly used to describe good programming practices.

DRY is not WET

DRY is my favourite: Don't Repeat Yourself... instead of writing the same code twice, wrap it up in a function and put it somewhere where it can be easily found and re-used.  This also means if you have to fix a problem, or improve your code, you'll only have to make your change once.

I tend to take DRY to the extreme, by writing little tools, either in PowerShell, or Python, or a little desktop app that takes care of any monotonous task that I might have. The tasks are usually things to do with my dev environment, like automatically powering up useful AWS resources when I login to Windows, and adjusting settings in JetBrains' Rider, so that it only has to compile the bare minimum it needs in order for me to debug the code I'm working on. Other tools I've written include swapping in and out fake and real libraries that my code connects to, so that I can debug under a variety of conditions. These tools not only save time, but allow me to focus on what I enjoy, and what really matters.

DROP is WET

I'd love to invent my own acronym here, DROP: Don't Repeat Other People.

Whenever writing your own generic code, there should be a few things that come to mind... Why has no-one else written this before? Oh, maybe they have. Is there a NuGet package I can reuse? Because, really, if it's generic, and your code is better than any other open source library out there, perhaps you should be either creating, or contributing to open source.

YAGNI... unless it's a kitten!

YAGNI means You Ain't Gonna Need It.  The theory is that your code will be simpler if you delete (or don't write) code that you may need in the future.  It also means you don't have to test it.

SOLID is five principles:

Chaos from multiple responsibilities

1. Single Responsibility: 

A class should only have one responsibility.

Example:  A Person class should not contain code that ensures the email address is in a correct, standard email address format.  Email validation is generic and should go elsewhere.

2. Open / Closed Principle:

Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.

To quote Robert C. Martin (Uncle Bob): 

When a single change to a program results in a cascade of changes to dependent modules, that program exhibits the undesirable attributes that we have come to associate with “bad” design. The program becomes fragile, rigid, unpredictable and unreusable. The open-closed principle attacks this in a very straightforward way. It says that you should design modules that never change. When requirements change, you extend the behavior of such modules by adding new code, not by changing old code that already works.[1]

This means, if you have created a class that is being used, don't make changes unless you're fixing bugs.  Inherit from it instead, and make the changes there. Of course all the cool kids are preferring composition over inheritance these days, so adding a property that's an object is fine, but work with abstractions. When I do this, I might use an interface like IContainX, and then implement a new interface IContainY. That way, your object can be extended with any existing code being referred to (X) remaining unchanged.

If you used TDD properly to build the class, and you're not actually changing the behaviour... You can make performance improvements as long as your original tests pass.

3.  Liskov Substitution Principle:

Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program.

So, if the Tesla class inherits from the Car class, you might not want to have a method Car.FillUpWithPetrol(), or you should simply not create a Tesla class inheriting from the Car class.

4.  Interface Segregation Principle:

Many client-specific interfaces are better than one general-purpose interface.

The reasons:

• Because any implementation should ideally implement every member.  When testing, and creating fakes, mocks or stubs, we don't want to have to care about whether or not a member is implemented if it's not related to the test.
• Unnecessary complexity (See my blog post on reducing complexity).  Not everyone who flies in a plane wants to see all the controls in the cockpit of a Boeing 747.  All they may need is a button to call the flight attendant.

5.  Dependency inversion principle:

Depend upon abstractions, rather than concretions.

This is often done using dependency injection. It's a way of loosely coupling objects. You don't solder wires from a lamp into a wall socket... you create interfaces, which both the lamp and source of electricity can use.

The way SOLID's used is a bit funny.  I say it's funny, because when I interview someone they usually remember SINGLE RESPONSIBILITY and waffle through the rest. Yup, we're only human. I've heard the word SOLID used a few times, when the speaker's intention was to refer to single responsibility. I don't think it really helps to ask about it at interviews, directly. It's possible that a good answer just shows that the person's attended many interviews. Perhaps a better test would be to offer some code, and ask what the interviewee would change about it. Although, when I've done that in the past, I tend to get a lot of silence... so who knows?

References:

1. Robert C. Martin "The Open-Closed Principle", C++ Report, January 1996

Microservices in a nutshell

 (or how to build a high performance distributed application / website that is easy to maintain)

Microservices at a high level is about splitting code horizontally based on context specific data models.  A single application could therefore consist of many services, each with its own business logic, data model and database.

There are drawbacks to creating Microservices.  The problem with popular patterns, is it's easy to consider them to be the right way to do things, but often the best approach is determined through experience, and considering what is really important for the specific context.  In many scenarios you will achieve better performance running code on a single computer, and you'll be able to develop it faster.  Nevertheless, for when it is a good approach, here are some advantages of splitting code into Microservices:

• All words in a model have a single meaning.  This avoids confusion and complexity which could come about from a large model.
• Less chance of code conflict: Development teams can focus on single services, without writing code that conflicts with teams working on other services.
• Services fail independently, rather than the entire system going down.
• Services can be written in different languages.
• A single service can be updated without affecting anything else.
• Enables continous delivery: services can be released independently when a feature is done, without waiting for a feature in another service.
• Easy to understand, as a small piece of functionality.  New developers do not need to understand the entire application.
• Scalability - easy to scale and integrate with 3rd parties.  Components can be spread across multiple servers.

Microservices can use event sourcing and CQRS to have optimized databases, so that any views which require extremely fast queries can have denormalized copies of the data, formatted specifically for their associated queries.

Event sourcing is a design pattern where changes to an application's state are stored as a sequence of immutable events, rather than directly modifying the current state. These events represent each action taken, allowing the system to recreate the current state by replaying the event log. This approach provides a complete audit trail and enables easy reconstruction or rollback of past states. It is commonly used in scenarios that require high levels of traceability and consistency.  A well known example of event sourcing is most source control software, which record the changes, rather than the state.

Some benefits of event sourcing are:

• It provides a 100% reliable audit log of the changes made to a business entity.
• One can get the state of an entity at any point in time by replaying events.
• Communication between services without too much concern about a service being unavailable.

CQRS stands for Command Query Responsibility Segregation.  CQRS means that instead of CRUD, where the same entity is updated via the same interface, the command to update the entity can be applied to a different model to the query to retrieve the information.

Some benefits of CQRS include:

• One can create a view database (readonly) combining everything for a particular UI view.
• Supports multiple denormalized views that are scalable and performant.

Microservices can be deployed in containers.

Some benefits of containers:

• Platform independence: Build once, run anywhere.
• Much smaller than VMs.
• Effective isolation and resource sharing.
• Speed: Start, create, replicate or destroy containers in seconds.
• Smart scaling: where you only run the containers needed in real time.
• Simpler to update OS for all containers than multiple VMs.
• Free from issues associated to environmental inconsistencies.
• Roll-out & roll-back with zero downtime.

High Precision Timing in SQL Server

Here's a simple snippet I wrote for high precision timing in SQL:

DECLARE @startTime datetime2;
DECLARE @endTime datetime2;

DECLARE @counter int = 0;
DECLARE @iterations int = 1000;

SET @startTime = SYSDATETIME();

WHILE @counter < @iterations
BEGIN
-- code to time here

SET @counter = @counter + 1;
END;

SET @endTime = SYSDATETIME();
SELECT DATEDIFF(MICROSECOND,@startTime,@endTime) / @iterations

Well Organized Unit Tests

Some people write unit tests by copying all of the arrange code to every unit test, so they will end up with code like this:

[TestMethod]

public void MyFunction_with_some_condition_does_something()

{

    // Arrange

    // 20 lines of setup code

    ...

    // Act

    ...

    // Assert

    ...

}

[TestMethod]

public void MyFunction_with_some_other_condition_does_something_else()

{

    // Arrange

    // The same 20 lines of setup code as above with one minor modification

    ...

    // Act

    ...

    // Assert

    ...

}

Of course the problem with this is that if you change one thing in your method's implementation, you may need to change every unit test, and it can get messy.  This makes people lazy, and they will neither want to make changes, nor fix their unit tests.  I'm not sure why people forget about good programming principles like DRY when it comes to unit tests.  So you could do something like this, and keep your unit tests small (3 or 4 lines each), easy to read and change:

private MyFunctionRequest myFunctionRequest;

private void Arrange_MyFunction(bool condition1 = false, bool condition2 = false)

{

    myFunctionRequest = new MyFunctionRequest

    {

      ...

    };

    if(condition1)

    {

      ...

    }

}

[Test]

void MyFunction_with_some_condition_does_something_else()

{

    // Arrange

    Arrange_MyFunction(condition2:true);

    // Act

    target.MyFunction(myFunctionRequest)

    // Assert

    ...

}

This looks okay, until you start to work with multiple methods in a class and find that you have to split your test class into regions and you have member variables that only pertain to a specific region.  Using regions is often an indication that your class is doing more than it should.  A better way to organise your tests is to have one class for each method being tested, like this:

[TestClass]

public class MyServiceTests

{

    protected MyService target;

    [TestInitialize]

    public void Init()

    {

        target = new MyService();

    }

    [TestClass]

    public class MyFunction_Method: MyServiceTests

    {

        private MyFunctionRequest request;

        private void Arrange(bool condition1 = false, bool condition2 = false)

        {

            request = new MyFunctionRequest

            {

                //...

            };

            if (condition1)

            {

                //...

            }

        }

        [TestMethod]

        public void With_some_condition_does_something()

        {

            // Arrange

            Arrange(condition1:true);

            // Act

            var actual = target.MyFunction(...);

            // Assert

            ...

        }

        [TestMethod]

        public void With_some_other_condition_does_something_else()

        {

            // Arrange

            Arrange(condition2:true);

            // Act

            var actual = target.MyFunction(...);

            // Assert

            ...

        }

    }

}

If your method being tested is particularly complicated, you could break it down even further, like this:

[TestClass]

public class MyServiceTests

{

    protected MyService target;

    [TestInitialize]

    public void Init()

    {

        target = new MyService();

    }

    [TestClass]

    public class MyFunction_Method: MyServiceTests

    {

        private MyFunctionRequest request;

        [TestInitialize]

        public void Init_MyFunction_Method()

        {

            request = new MyFunctionRequest();

        }

        [TestClass]

        public class With_some_condition: MyFunction_Method

        {

            [TestMethod]

            public void Does_something()

            {

            }

        }

    }

}

In Visual Studio, your solution explorer will look like this:

C# In Memory Search Performance Comparison

Just in case one ever wants to do really fast in memory searches, it's useful to know which classes are the fastest, so I did some tests.  Here are the results from searching for a million integers in an array of a million integers.

Warming up...

Single threaded search...

Testing HashSet...
Create (and sort) took 209ms
Search took 208ms
Total time 417ms

Testing Dictionary...
Create (and sort) took 375ms
Search took 117ms
Total time 492ms

Testing BinarySearch...
Create (and sort) took 187ms
Search took 554ms
Total time 741ms

Testing SortedList...
Create (and sort) took 419ms
Search took 543ms
Total time 962ms

Testing HashTable...
Create (and sort) took 1180ms
Search took 530ms
Total time 1710ms

Multithreaded search...

Testing HashSet...
Create (and sort) took 114ms
Search took 65ms
Total time 179ms

Testing Dictionary...
Create (and sort) took 218ms
Search took 55ms
Total time 273ms

Testing BinarySearch...
Create (and sort) took 186ms
Search took 168ms
Total time 354ms

Testing SortedList...
Create (and sort) took 431ms
Search took 171ms
Total time 602ms

Testing HashTable...
Create (and sort) took 972ms
Search took 343ms
Total time 1315ms


And here's the source code I used to get these stats:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using System.Threading.Tasks;
using System.Collections;

namespace PerformanceTesting
{
    class Program
    {
        static void Main(string[] args)
        {
            Random rnd = new Random();
            var warmUp = Enumerable.Range(0, 1000000)
                .Select(i => rnd.Next())
                .Distinct()
                .Take(100000)
                .ToArray();
            var randomIntegersToStore = Enumerable
                .Range(0, 10000000)
                .Select(i => rnd.Next())
                .Distinct()
                .Take(1000000)
                .ToArray();
            if (randomIntegersToStore.Count() != 1000000)
                throw new NotSupportedException("Incorrect number of integers");
            var randomIntegersToSearchFor = Enumerable.Range(0, 1000000)
                .Select(i => rnd.Next())
                .ToArray();
            var tests = new List<ITest>
            {
                new HashSetTest(),
                new DictionaryTest(),
                new BinarySearchTest(),
                new SortedListTest(),
                new HashTableTest()
            };
            Console.WriteLine("Warming up...");
            Console.WriteLine();
            foreach(var test in tests)
            {
                test.Initialize(warmUp);
                test.Search(warmUp, false);
            }
            Console.WriteLine("Single threaded search...");
            Console.WriteLine();
            RunTests(tests, warmUp, randomIntegersToStore, randomIntegersToSearchFor, false);
            Console.WriteLine("Multithreaded search...");
            Console.WriteLine();
            RunTests(tests, warmUp, randomIntegersToStore, randomIntegersToSearchFor, true); Console.ReadLine();
        }

        private static void RunTests(IEnumerable<ITest> tests,
            int[] warmUp, int[] randomIntegersToStore, int[] randomIntegersToSearchFor,
            bool multithreadedSearch)
        {
            foreach (var test in tests)
            {
                test.Initialize(warmUp);
                Console.WriteLine("Testing " + test.GetType().Name.Replace("Test", "") + "...");
                var stopwatch = Stopwatch.StartNew();
                test.Initialize(randomIntegersToStore);
                stopwatch.Stop();
                var createTime = stopwatch.ElapsedMilliseconds;
                Console.WriteLine("Create (and sort) took " + createTime + "ms");
                test.Search(warmUp, multithreadedSearch);
                stopwatch.Restart();
                test.Search(randomIntegersToSearchFor, multithreadedSearch);
                stopwatch.Stop();
                var searchTime = stopwatch.ElapsedMilliseconds;
                Console.WriteLine("Search took " + searchTime + "ms");
                Console.WriteLine("Total time " + (createTime + searchTime).ToString() + "ms");
                Console.WriteLine();
            }
        }

        interface ITest
        {
            void Initialize(int[] randomIntegers);
            void Search(int[] randomIntegers, bool multiThreaded);
        }

        class HashSetTest : ITest
        {
            private HashSet<int> hashSet;
            public void Initialize(int[] randomIntegers)
            {
                hashSet = new HashSet<int>(randomIntegers);
            }

            public void Search(int[] randomIntegers, bool multiThreaded)
            {
                if (multiThreaded)
                    Parallel.ForEach(randomIntegers, i => hashSet.Contains(i));
                else
                    foreach (var i in randomIntegers)
                    {
                        hashSet.Contains(i);
                    }
            }
        }

        class BinarySearchTest : ITest
        {
            private List<int> list;
            public void Initialize(int[] randomIntegers)
            {
                list = new List<int>(randomIntegers);
                list.Sort();
            }

            public void Search(int[] randomIntegers, bool multiThreaded)
            {
                if (multiThreaded)
                    Parallel.ForEach(randomIntegers, i => list.BinarySearch(i));
                else
                    foreach (var i in randomIntegers)
                    {
                        list.BinarySearch(i);
                    }
            }
        }

        class SortedListTest : ITest
        {
            private SortedList<int, int> list;
            public void Initialize(int[] randomIntegers)
            {
                list = new SortedList<int, int>(randomIntegers.ToDictionary(i => i));
            }

            public void Search(int[] randomIntegers, bool multiThreaded)
            {
                if (multiThreaded)
                    Parallel.ForEach(randomIntegers, i => list.ContainsKey(i));
                else
                    foreach (var i in randomIntegers)
                    {
                        list.ContainsKey(i);
                    }
            }
        }

        class DictionaryTest : ITest
        {
            private Dictionary<int, int> dictionary;
            public void Initialize(int[] randomIntegers)
            {
                dictionary = randomIntegers.ToDictionary(i => i);
            }

            public void Search(int[] randomIntegers, bool multiThreaded)
            {
                if (multiThreaded)
                    Parallel.ForEach(randomIntegers, i => dictionary.ContainsKey(i));
                else
                    foreach (var i in randomIntegers)
                    {
                        dictionary.ContainsKey(i);
                    }
            }
        }

        class HashTableTest : ITest
        {
            private Hashtable hashTable;
            public void Initialize(int[] randomIntegers)
            {
                hashTable = new Hashtable(randomIntegers.ToDictionary(i => i));
            }

            public void Search(int[] randomIntegers, bool multiThreaded)
            {
                if (multiThreaded)
                    Parallel.ForEach(randomIntegers, i => hashTable.ContainsKey(i));
                else
                    foreach (var i in randomIntegers)
                    {
                        hashTable.ContainsKey(i);
                    }
            }
        }
    }
}