OpenCover: ⟳ code coverage metrics with CI build

OpenCover analyzes your .NET codebase and generates an XML report rich with detail about the extent and quality of your code coverage. You think you’ve covered your testing bases, but how do you know? What if there’s actually a sneaky runner leading off second base? If so, OpenCover will blow his cover wide open. In this post I will show you how to retrofit your automated build with OpenCover. Then we’ll make the output more human-readable with Report Generator.

If you read my article about Psake , you’ll recall that I started the open source project Resfit in order to experiment with Acceptance Test Driven Development. I started the project with NCrunch and though the 30 day trial ran out, my appreciation for NCrunch’s coverage metrics did not. I found OpenCover, which generates some amazing code coverage metrics. Only trouble is, they’re all in XML - Better suited to be read by a build server than a lowly Developer like myself. That’s where ReportGenerator comes in, which “converts XML reports generated by OpenCover, PartCover, Visual Studio or NCover into human readable reports in various formats”.

Retrofitting your build tasks with OpenCover

First add the OpenCover NuGet package to your project. Then, at a high level, you’re going to modify your continuous integration build script to run your tests through OpenCover rather than NUnit. OpenCover itself will take care of interacting with NUnit. In this example we will be modifying a Psake build script, but the process will be similar with other approaches.

For reference, here are the PowerShell files we’ll be most interested in from the Resfit project:

• psake_helpers.ps1
• default.ps1

Creating a wrapper function to call OpenCover

psake_helpers.ps1 contains the helper function, Run-Tests, which takes care of the nitty gritty details involved in calling OpenCover. Its implementation looks like this:

Function Run-Tests {
	[CmdletBinding()]
	Param(
		[Parameter(Position=0,Mandatory=1)]$openCoverExe,
		[Parameter(Position=1,Mandatory=1)]$testRunner,
		[Parameter(Position=2,Mandatory=1)]$testRunnerArgs,
		[Parameter(Position=3,Mandatory=1)]$coverageReportPath,
		[Parameter(Position=4,Mandatory=1)]$filter,
		[Parameter(Position=5,Mandatory=0)]$excludeByAttribute,
		[Parameter(Position=6,Mandatory=0)]$excludeByFile
	)

	Exec {
		& $openCoverExe `
			-target:$testRunner `
			-targetargs:$testRunnerArgs `
			-output:$coverageReportPath `
			-register:user `
			-filter:$filter `
			-excludebyattribute:$excludeByAttribute `
			-excludebyfile:$excludeByFile `
			-skipautoprops `
			-mergebyhash `
			-mergeoutput `
			-hideskipped:All `
			-returntargetcode
	}
}

One thing you’ll notice here is that all of the arguments to the OpenCover executable are lower case. Don’t make the mistake I did and try to camelcase them. The arguments must be lower case. Crazy thing is, OpenCover doesn’t complain – it just ignores the arguments it doesn’t recognize. You’re left scratching your head wondering why your filter isn’t working.

Adding the Psake build task

Now let’s focus on the Psake task responsible for running the unit tests. It’s in default.ps1.

Task UnitTests `
	-description "Run all unit tests" `
	-depends Build `
	-precondition { $(Get-ChildItem -Path $outputDirectory *.Tests.dll).Count -gt 0 } `
{
	$assemblies = Get-ChildItem -Path $outputDirectory *.Tests.dll `
		| ForEach-Object { Quote-String($PSItem.FullName) }

	$testResultsXml = Quote-String("$testResultsDirectory\{0}Results.xml" -f $Task.Name)
	$testOutput = $testResultsXml -replace 'xml','txt'

	$nunitArgs = "$assemblies /result:$testResultsXml /out=$testOutput /noheader"

	Run-Tests -openCoverExe $openCover `
			  -testRunner $nunit `
			  -testRunnerArgs $nunitArgs `
			  -coverageReportPath $testCoverageReportPath `
			  -filter $testCoverageFilter `
			  -excludeByAttribute $testCoverageExclusionAttribute `
			  -excludeByFile $testCoverageExcludeFiles
}

Notice that a task precondition ensures that test assemblies were created during the build. Then we get a list of the test assemblies which will be passed to NUnit by OpenCover. Since OpenCover will be calling NUnit for us, we have to surround a number of the arguments with two double quotes. That’s what that Quote-String function does for us. It’s implementation is also in psake_helpers.ps1.

As you read through this function, you notice that not all of the variables are declared here. Some very important parameters to OpenCover are constructed in variables declared in the properties array at the top of the default.ps1 file.

$testCoverageFilter = Quote-String("+[*]* -[*.Tests]* -[*.Requirements]*")
$testCoverageExclusionAttribute = "System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute"
$testCoverageExcludeFiles = "*\*Designer.cs;*\*.g.cs;*.g.i.cs"

Selecting which assemblies to include with the coverage filter

The coverage filter is super important and determines which of your assemblies will be examined by OpenCover. Its syntax is a little odd at first but not too difficult. The schema is (+|-)[AssemblyFilter]TypeFilter. “+” means include. “-” means exclude. So in the code sample above, +[] tells OpenCover to include all assemblies and classes. But that is followed by -[.Tests] and -[.Requirements], which tell OpenCover to ignore assemblies ending with “.Tests”. We don’t usually write tests for our tests, so there’s no point in seeing how much test coverage our tests have.

Excluding a class from code coverage

The $testCoverageExclusionAttribute is useful when you want to exclude a particular class from your code coverage metrics for some reason. Here we used ExcludeFromCodeCoverageAttribute which is located in the System.Diagnostics.CodeAnalysis namespace. Then all you have to do is decorate your class with [ExcludeFromCodeCoverage] and OpenCover will ignore it.

Excluding files from code coverage

$testCoverageExcludeFiles is handy for excluding certain files from code coverage metrics. In this case we are excluding files with extensions that indicate generated code, such as .g.cs and .g.i.cs. Generated code doesn’t require test coverage because you already wrote extensive unit tests for the code generation process itself. Right? :)

Make your OpenCover results human-readable with ReportGenerator

Now that you’re generating test coverage metrics, how do you read them? A glance at the generated xml file reveals… well, xml. It’s possible to read it, just like it’s possible to read assembly language, but you can think of better uses for your time. What you need is ReportGenerator . ReportGenerator takes the XML code coverage report as input and spits out a nicely formatted HTML report. It even includes little red and green bar graph graphics which make areas in need of coverage bleedingly obvious. For example, Resfit’s report indicates that ListDecoratorBase could use some additional coverage. I happen to know that that class doesn’t need much unit test coverage, so it might be a good candidate for the ExcludeFromCodeCoverage attribute.

Concluding remarks

So there’s OpenCover in a nutshell. On one hand it doesn’t provide the same instant gratification that tools like NCrunch are dealing, but it is easy to integrate into your Continuous Integration pipeline. With tools like Coveralls you can even fail the build if code coverage drops below a predetermined threshold. In a later post, I’ll show you how to add cool badges to your GitHub project, including one that proclaims your project’s percentage of coverage. Pretty cool stuff, my friend. Feel free to leave your comments below.