Silverlight Unit Test Applications and CI

After getting the first test working, I now want to get a continuous integration (CI) system in place to keep me honest. I'm not sure if Microsoft offer anything within TFS for automated Silverlight builds, but for EasyPlayer I'm using GitHub for source control and as I'm only using the free tools from MS (e.g. Visual Web Dev 2010 Express), that would rule out TFS.

Looking at the options, there's CruiseControl.NET which I've used heavily in the past, TeamCity and Jenkins (formally Hudson). There are also some interesting cloud-based solutions (e.g. AppHarbor) but I think these may be for websites only.

Hudson vs. Jenkins

I do like CruiseControl.NET but it is very XML heavy, so I'm going to look at alternatives this time. I've used Hudson in the past and this was pretty nice - the web UI is great and configuration is straight-forward. I believe it can also be scripted so you could automate project configurations for example.

Greatly summarising, Hudson is an open source CI tool which came out of Sun and after the takeover by Oracle there was a trademark issue regarding the Hudson name, so the project forked into Jenkins. Hudson is still around, but it seems the momentum is with Jenkins.

The Setup

It was quite easy to setup Jenkins - just follow the guide to install Jenkins as a Windows service. To build the solution, we need the GIT, GitHub, MSBuild, and NUnit plugins, all available from the Jenkins Plugin Manager.

Jenkins Plugin Setup

The configuration of these is pretty self-evident: GIT needs to know the location of git.exe and MSBuild needs the location of msbuild.exe (C:\Windows\Microsoft.NET\Framework{version}\msbuild.exe).

Next, the new project setup is equally trivial:

Jenkins Project Setup

Essentially, we need the Git repository (git://github.com/andywhitfield/EasyPlayer.git) and the MSBuild arguments (EasyPlayer.sln solution file and /t:Build target).

Save the new project configuration and run it. It should build the project successfully:

Jenkins First Run

No Tests

There is one problem: we're not actually running any tests. This is where we need the help of a project called StatLight. This will run the tests and output the results to a file, similar to what nunit-console.exe would do.

For simplicity, the StatLight.exe is included in the project, so all that's needed is to add a command to the build steps:

Jenkins StatLight Setup

And it doesn't work. StatLight executes but doesn't report any tests have run.

Interactive Tests

The problem is that Jenkins is running as a Windows Service so StatLight tries to launch a browser to run the Silverlight test application but services can't, by default, launch any UI elements. The solution is simple enough: just go into the services control panel, and allow it to interact with the desktop:

Jenkins Service

Re-start the Jenkins service and re-run the build. This time we should see some results in the file StatLight generated:

Jenkins StatLight Result File

Yes! A successful test.

Reporting Test Results

Clearly it'd be better to have the results displayed right on the Jenkins project page, rather than having to look in some random file. This is where it got a bit tricky! Jenkins does support NUnit - as mentioned above, I recommend the NUnit plugin. However, the StatLight output isn't in the NUnit format. Its close, however, so perhaps we just do a little conversion of the StatLight output into an NUnit format and the Jenkins plugin will simply do its thing and report the test results.

The conversion is simply one xml format to another, so the first thing that comes to mind is xslt. However, for me this is a bit like the quote about regular expressions:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

I just know it'd take me a silly amount of time to do this, when I could just use C# to read the original, and write out a new xml file. So that's what I did.

Again I just put the binary into the lib folder and in Jenkins and another task to call the converter passing in the StatLight file and the destination output file:

Converting StatLight output into NUnit format

Also, in the above, NUnit is configured to pick up the results file.

Re-running, Jenkins now reports my one successful test. Magic.

Now all that's in place, I can re-focus back on EasyPlayer itself and start getting to the point where I can play a downloaded MP3 file!


Comments:

That's right - I believe StatLight can now output a nunit results file directly so no need for any post-processing. Although I've not actually tried it so unfortunately can't help with why Jenkins isn't showing the individual tests.

Actually, it looks like StatLight now supports various output formats including TRX, so transformation isn't necessary?

I'm using it here, and it works... except my Jenkins Test Results report shows only a single package named "(root)" and not the individual tests (though my test count is accurate).

Nice work! You should wrap this up into a StatLight plugin for Jenkins. I know at least one person who would use it!


Comment Guidelines
See the FAQ for details on the full rules and guidelines. No Spam. Write clearly and thoughtfully - no bad language.