AutoTest.Net 1.2 is out containing a Visual Studio Addin!

We are proud to announce release 1.2 of AutoTest.Net today. In this version we have pulled back all the Visual Studio Addin functionality from the ContinuousTests (Mighty-Moose) source into AutoTest.Net. This means that you will get all test and build feedback right into a feedback window in Visual Studio. This speeds up the work flow drastically from using AutoTest.Net through the winforms app or console. In addition you will have features like manually running tests from right-click in code, solution explorer and class view. You will also be able to debug failing tests from the feedback window or directly from right-clicking the test.

Getting started
To get started goto https://github.com/continuoustests/AutoTest.Net/archives/master and download the AutoTest.Net-v1.2.0.exe, run the installer and then open a solution in Visual Studio. By default AutoTest.Net starts paused when used as an addin so it will not detect file changes right away. You can still do manual test runs though. The feedback will arrive into the feedback window which you can open through AutoTest.Net->Feedback Window. To start detecting changes click AutoTest.Net->Resume Engine. Now just modify a file/files and hit save and things should start happening. If you want to have AutoTest.Net enabled by default go to Global Configuration and change StartPaused to false (when changing the configs global or solution you need to run AutoTest.Net->Restart Engine for them to take effect).

Work flow
When working with ContinuousTests or AutoTest.Net you no longer have to wait for your build or test runs to complete. This changes your work flow radically. At first it will be hard to let go of old habits but after a little while not waiting for the build/test result becomes natural. Most of the time when you build and run tests the outcome is what you expect so there is no need to wait for it. Since the feedback loop will be shortened the likelihood of this increases. If you get an unexpected outcome you are only a couple of ctrl+z/commenting away from previous state anyway.
There are a few short-cut keys that you will use very frequently.

• Ctrl+Shift+J => Open/Toggle Feedback Window
• Navigate Feedback Window with arrows or vim keys (j,k)
• Hit enter in the list to go to build error or failing test
• Hit the letter I to open the information window for extended information
• Hit the letter D when positioned on a test to debug it
• Ctrl+Shift+Y, A => Build solution and run all tests
• Ctrl+Shift+S => Save all files in Visual Studio

The first thing you do when opening a solution is Ctrl+Shift+Y, A (Build solution and run all tests) to get current state of the system. From now on the feedback window will work as current system state. The status text in the window represents last action while the content of the list represents current state. A failing test will stay in the window until it’s fixed even though current test run succeeded (ran tests in another assembly for instance).
From now on you work incrementally: Write code, Ctrl+Shift+S, Continue writing code, Fail, Ctrl+Shift+J and navigate to to failure, Jump to code/ Debug, Write code, rinse repeat, rinse repeat. In that way you always work on emptying the feedback window and when it’s empty you write failing tests to fill it back up.

We hope you will have fun with this product and don’t hesitate to report issues on the github page https://github.com/continuoustests/AutoTest.Net. We love reported issues as it’s what makes the product better. If the issue is followed by a pull request we are talking total awesomeness.

AutoTest.Net no longer in beta!

Finally AutoTest.Net has reached it's first official release. There has been tons of contributions since last beta release and most of it has been coming from the Mighty-Moose project (www.continuoustests.com). Amongst others AutoTest.Net now has it's own test runner that hosts NUnit and XUnit (more to come). This means that there is no longer need to manually configure these test runners. Also the configuration as it is today is set to 64bit/Visual Studio 10. So if that's your setup no configuration needed for MSBuild or MSTest either.

New features

• New hosted test runner (AutoTest.TestRunner)
• MSpec is now supported thanks to contributions from Alexander Groß himself. For now it has to be manually configured but it's planned to get it supported in our hosted runner.
• Support for either watching a directory or a solution
• When watching a solution you can choose to build the solution instead of individual projects
• Setting for running failed tests first

For a direct link to the the new version try this one https://github.com/downloads/acken/AutoTest.Net/AutoTest.Net-v1.1.zip.

From what I see there are actually quite a few users out there. Still there is not too much feedback. There have been some talk about both the WinForms and the Console UI. If you have questions or want to discuss join our group http://groups.google.com/group/autotestnet. And I will encourage anyone to report anything they want considered changed / added to the issues list on github (https://github.com/acken/AutoTest.Net). Let's together make this an awesome piece of software!

-Svein Arne

Testing through interfaces

When at Vagif Abilov's BBQ while talking about testing Greg Young said something like "I wonder what tests would look like if we wrote tests against interfaces instead of their implementation". And really, what would they look like?

The scenario was that you have an interface that has multiple implementations and you would want to write tests against the interface testing all implementations. This would seriously reduce the number of tests you would have to write. So let's give it a try.

The first thing we need to keep in mind is that we're writing the tests on something abstract. This means that we don't really know what to expect. When passing x in to the various implementations it's not certain that all of them will answer y. Actually hopefully only one of them would answer y where x is act and y is assert. If not there would be multiple implementations doing the exact same thing and that would kind of defeat the purpose. Off the top of my head that leaves us with the following scenarios:

X stays the same while Y varies
This would be something like an ICalculator. The ICalculator would have implementations like DecimalCalculator and OctalCalculator. When running tests here we would end up with results like this:

• DecimalCalculator: 7*7 = 49
• OctalCalculator: 7*7 = 61

Which means that when writing these types of tests we need to be able to handle asserting on certain values pr. implementation.

X varies while Y stays the same

Let's imagine that we have some type of parser taking xml in returning a list of objects of a certain type. This would typically mean one implementation pr. xml schema while the output could be the same. So writing these types of tests  we'll have varying code for passing parameters while the assert would stay the same.

Ok, that wasn't too bad. We could probably make this look clean. Now over to some other aspects that we'll have to deal with.

Dependencies

With the right (wrong) implementation faking dependencies might be a hellish thing with this solution. I guess that's a good thing as it forces us to not make a mess of it. But still we need some way of handling setting up dependencies for the implementations.

Resolving implementations

We need a way to retrieve all implementations for an interface. Of course this is something we do all the time with DI containers so any DI container would provide us with what we need here. We could probably do something smart here to inject the faked dependencies we'll need for each implementation.

With this in mind let's set up a test for the calculator scenario. The first thing I did was creating a class for handling the plumbing. Right now this class takes care of resolving all implementations of the chosen interface, running the test on each implementation and performing specified assertions. My test ended up looking like this:

        [Test]
        public void Should_multiply()
        {
            var tester = new InterfaceTester<ICalculator>();
            tester.Test(c => c.Multiply(7, 7))
                .AssertThat<DecimalCalculator>().Returned(49)
                .AssertThat<OctalCalculator>().Returned(61);
        } 

I'm quite happy with that. This test is both extend able and readable. Now let's do the same to the scenario with the string parser. I'll just extend the plumbing class used in the previous example to handle varying input parameters. The implementation ended up looking like this:


        [Test]
        public void Should_parse_number()
        {
            var tester = new InterfaceTester<INumberParser>();
            tester
                .Test<XmlParser>(x => x.Parse("<number>14</number>"))
                .Test<StringParser>(x => x.Parse("14"))
                .Returned(14);
        }


I can't say I'm as happy with this one as the complete delegate is copied for both implementations and not just the part that differs. But still it's a huge simplification compared to writing a full test suite pr implementation.

I guess I'll leave it at that for now. What this does not cover is setting up dependencies which likely will complicate the implementation a bit. After doing this implementation I can really see the value of writing my tests like this. It would save me time and energy and would leave me with a cleaner simpler test suite. The implementation ended up being fairly simple. Initial conclusion: Writing tests against interfaces is a good idea!

I'd love to hear  your thoughts on this! And if you're interested in the full source code let me know and I'll upload it to github or something.

Tooling visions

Lately I have felt more and more uncomfortable about the tooling I'm currently working with. I feel a lot of the tools are not helping me reaching my goal. Frankly their in my way. The whole thing started when I started using ReSharper and saw what an IDE is about. ReSharper truly helps you accomplish things. It keeps you focused on your real goal: producing quality code. The only bad thing about ReSharper is that it's tied to Visual Studio. Visual Studio has become a horrid beast of an application. It's packed with features that to me has nothing to do with the application I use to write code. It doesn't help me any more than it's in my way.
My second wakeup call was when I started using Git. Specially after watching Linus Torvalds talking about he's ideas behind it and why he made it like he did. One of the reasons he says was to create a source control system that does it's job well and doesn't get in your way. And he succeeded! When working with Git you have to deal with it when you pull, commit or push. And those are the times you're supposed to deal with source control. You shouldn't have to deal with source control because you want to edit a file or having the source control system insert padlocks left right and center. You shouldn't have to think about whether your computer is online or offline when you work with your code. Going back to working with TFS made me realize how much time I waste using a tool. Time that I could have spent solving real problems.

Ok, that was the venting part :) Now to something a little more constructive. I have been thinking about how my ideal set of tools would look and what would be important. There are some points I really want to focus on. The first point applies as well to code: SEPARATION OF CONCERNS! Each tool should help you solve a single problem and it should do it well, extremely well. Second, as mentioned it should stay out of your way. It should know what it's trying to help you solve and act like that twin that completes your sentences. Third, it should not compromise. When you work with a mess of a project having everything depending on everything it should be painful. The tool should keep it's focus on helping you produce quality code and not make compromises to help dealing with a festering pile like Uncle Bob puts it.

First off let's deal with what we call the IDE. To me an IDE is notepad+ReSharper+navigation, and that's what I think it should be. It should be there to help us produce quality code as efficiently as possible providing intellisense, auto complete, refactoring and everything that has to do with writing code. And that to me has nothing to do with building binaries, running tests, debugging and deploying. Though I understand why IDE's have ended up where they are it's time to move on. We're no longer hacking and hoping. We don't set breakpoints and step through half the application as part of our work pattern. We write code and watch tests fail and pass. To me the IDE is about efficiently writing code.

Of course we need to compile and run tests and that should be it's own tool. We already have continuous testing tools like JUnitMax, Ruby's autotest and AutoTest.NET which I'm currently working on (add cheesy commercial part here). This tool should basically stay out of your way. The only time we would want to interact with this tool is when we have broken something. It should build and run only what it needs to and only grab our attention when something has gone wrong. This is the tool that would bind the editor and the debugger together. When something has gone wrong we should be able to get the right and enough information. When builds or tests fail we should be able to easily move to the right file and position in the editor to fix whatever is wrong.

Now to the debugger. The way I see it debuggers the way they work today are optimized for full system debugging not the simple "now what the heck did I just do to fail this test". And that's what I'm looking for 95% of the time. For these types of tasks I don't think that debugging through the IDE is helping. I don't think displaying the code file by file, class by class, function by function like it's written is the best way. And certainly not stepping through it. Something I do think would be more efficient is analyzing a series of snapshots showing where the execution had it's turning point, what threw an exception and things like that. I have tons of idea's that I'm hoping to realize through the ClrSequencer project. I think I'm going to dive a little bit deeper into this in another blog post.

I guess that's enough rambling for tonight. It's probably not the last thing you'll hear from me on the subject. Tooling is very important and tooling should help you not fight you and I have been feeling a lot of the latter lately.

Continous testing with AutoTest.Net

It's about time for me to do some writing about AutoTest.Net. This is a project I have been working on for the last 2-3 months. It's a continous testing tool for .Net originally based on ruby's autotest. After playing with Ruby for a couple of evenings I really enjoyed the way you could work with it. My usual work circle of "write code, build, wait, run tests, wait" was replaced with "write code, save". Luckily the code I write tends to work more often than it breaks so waiting for builds and tests to run is a waste of time. Specially when I build and run tests about every 30 seconds or so.

So after the joy of working like that in Ruby I was determined to find a tool like that for .Net. After a bit of searching I found AutoTest.Net. The project was hosted at code.google.com and was initiated by James Avery but because of not having enough time on his hands the project was put on hold. I really wanted a tool like that so I went and got his permission to continue the project. It's now hosted on github.com/acken/AutoTest.Net. Today it contains support for both .Net and mono and it's cross platform. NUnit, MSTest and XUnit are the testing frameworks supported today and MbUnit will be added soon. It supports running tests from multiple testing frameworks in the same assembly.

Now, how does it all work you say? The whole thing consists of a console application and a winforms application. By now I use the winforms app about 98% of the time so that's what I'm going to show here.

The first thing you do ofcourse is to go to this link and download the latest binaries and unzip them to the folder of your choice. Locate the file named AutoTest.config and open it in your favorite xml editor. Now let's edit a few settings:

1. DirectoryToWatch: Set this property to the folder containing the source code you want to work with. AutoTest.Net will detect changes to this folder and it's subfolders.
2. BuildExecutable: This is the path to msbuild or in mono's case xbuild. You have the possibility to specify a version of msbuild pr. framework or visual studio version.. For now let's just specify the default <BuildExecutable> property. Something like C:\Windows\Microsoft.NET\Framework\v3.5\MSBuild.exe.
3. Now let's specify a testing framework. You can pick from zero to all though zero wouldn't be any fun. I'll go with NUnit in this example.
4. The last thing we want to do is to specify a code editor (<CodeEditor>). Let's pick visual studio. We can pass visual studio the file to open and the line to go to. Sadly there's a bug in visual studio preventing it to go to the right line :( So for now we'll rely on Ctrl+G. Anyways the config has visual studio set up correctly by default. Just make sure the path to devenv.exe is the same as on your machine.

Now we're ready to start the AutoTest.WinForms.exe application and do some real work.The first thing you 'll see is a form looking like this.

The only interesting thing right after startup is the button in the top right corner. As you can see now (gonna do something about the colors) the button is yellow. Behind this button you'll find the status for AutoTest.Net application. It's yellow now because the configuration has generated a warning. If the button is red an error has occurred within AutoTest.Net. Right now the window will look like this.

So let's go ahead write some jibberish in one of the files inside the folder we're watching and save the file. AT.Net should start working right after you save the file.

And of course jibberish means errors which will result in this.

When selecting one of the lines in the list you'll get the build error/test details underneath and you can click on the links to open the file in visual studio. Now let's fix the error we just made and save the file and let's see what happens.

And as expected it goes green with 5 succeeded builds and 221 passed tests. That's basically it. From here on it's lather, rinse, repeat.

Right now it's in alpha and it will of course have some bugs here and there. I hope this post will tempt you to try it out. Even at an early stage like this it's a really effective way of working!

(Red-Green)N-Refactor

Red/Green/Refactor is the TDD mantra. And fair enough, it's a good description of the concept of TDD which I guess it's meant to describe. Heck it even describes the order you should perform these steps in when writing a test. Make sure the test is able to fail, make sure it passes for the right reasons and when done clean up. This is all good but I see people being mislead by this mantra. To explain this a bit better let's have a look at Uncle Bob's three rules of TDD

1. You are not allowed to write any production code unless it is to make a failing unit test pass.

2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.

3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

Following these rules (and you should!) you'll see that when writing code you're mantra should in fact be red/green/red/green/red/green/............./refactor. Frankly though that wouldn't make a very good mantra. At the same time red/green/refactor doesn't work as an implementation pattern. The same way as agile methods will have you delivering working software in short iterations these three rules will deliver a suite of passing tests in about 30 second long iterations. And more importantly every red/green cycle will have you focusing on a single small task which is an efficient way of working. Summing up the mantra through Uncle Bob's three rules we get red/green repeated until the test is completed. This gives us (Red-Green)N.
So, why isn't refactoring part of the three rules of TDD? TDD or not TDD you should ALWAYS refactor. Make sure your code is readable and maintainable. TDD just happens to make your code a lot easier to refactor safely. Given that the concept of TDD tells you to refactor for every completed test (and you should!) the mantra you should follow when implementing is (Red-Green)N-Refactor.

Solution structuring and TDD in Visual Studio

I'm currently working on solution structuring for a large system using TDD. To be able to work efficiently within the project we have defined a set of solution types with different purposes. This is the setup we ended up with:

Workbench
TDD requires that the solutions you spend most of your time in builds as fast as possible. To achieve this you have to keep the project number within the solution to a minimum. If you cannot get arround having multiple dependencies for every project, binary references would be the way to go. We have choosen a more decoupled approach where every project depends upon abstractions and interfaces are wired to implementations through a DI container. Contracts and interfaces are separated out into contract projects. This keeps project references down to just referencing interface/contracts projects. The build output from this solution would not be able to run since the contract implementations are not referenced here. The solution contains all environment independent unit and integration tests for this workbench's projects. Because of practicing TDD it's not important that the solution is able to run but that the tests are able to pass. A workbench exists for every bounded context and standalone code library.

Test rig
Even though most of our time is spent writing tests and code in the workbench solutions we some times have the need to debug a running system. These solutions contain all code needed to run parts of the system like hosting services or even complete running systems. The test rig solutions are usually quite large and takes time to build but, theire there for us to ocasionally test the system locally.

Continous Integration solution
This solution contains all projects and  all environment independent unit and integration test for the complete system. This solution is part of the continous integration build performed on check in. Naturally CI runs all tests on every build.

System test solution
We need a solution containing environment dependent tests requiring things like database access or a running system.The tests within this solution should run on every deployment to the test environment to make sure the system wiring is intact.

As for deployment both test rigs and continous integration solutions contains enough of the system to be able to perform deployment.

It would really be interresting to see how other people are working on similar projects.