Reference

Advanced Tests

Add-In Interface Options TestRunner Export Results

Advanced Assertion Topics: Severity levels
Local Severity Levels
Stopping tests
Test-Resource Management: Fixture Lifetime
Error handling for Create, Setup, TearDown, and Destroy
Loading external files
Manual fixtures (without reflections)

Advanced Assertion Topics

Severity levels

This version of vbUnit introduces severity levels for assertions. This allows you to determine the effect of a failed assertion. In previous versions of vbUnit, a failed assertion would have no effect on subsequent assertions, while an error would exit the current test method and continue in the next one. This default behavior has now been changed such that a failed assertion will exit the current test method. Execution will then continue in the next test method. Hence, a failed assertion is now treated like an error.

If this conflicts with your needs, Don't Panic! The new method IAssert.SetSeverity allows you to change this behavior:

Sub IAssert.SetSeverity(Severity As vbUnitScope, Optional Duration As vbUnitScope = vbUnit_Method)

This method takes two arguments of the the type vbUnitScope:

Public Enum vbUnitScope
    vbUnit_Assertion = 0
    vbUnit_Method = 1     'this is the default
    vbUnit_Fixture = 2
    vbunit_Suite = 3
    vbUnit_AllTests = 4
End Enum

The severity level determines the effect of a failed assertion. In vbUnit2, each assertion had severity 0, which means that a failed assertion would have no further effect. The default severity level in vbUnit3 is now 1 (vbUnit_Method). This means that a failed assertion will break out of the current testmethod and execution will continue at the next testmethod. For severity 2 (Fixture), a failed assertion will stop execution in the current fixture, severity 3 will break out of the current suite, and severity 4 will stop all further tests.

For example, if you acquire a complicated resource without which all further tests in a given fixture would be meaningless, set the severity to vbUnit_Fixture before checking if the resource was acquired sucessfully:

'Critical Initialization Test

'in Fixture class TestMyExpensiveResource.cls:
Public Sub TestInitialization()
   m_assert.SetSeverity vbUnit_Fixture 'a failed assertion in  
                                       'this method will stop  
                                       'testing in this fixture
   m_assert.Verify m_resource.IsValid
End Sub

Public Sub TestUseResource()
   'resource is valid, we can use it now!
   '...
End Sub

The second, optional parameter of SetSeverity determines the duration of the severity assignment. When execution leaves the scope of the duration, the severity level goes back to the default. The duration level 0 applies only to the next assertion, duration 1 applies to all assertions in the current method that come after the SetSeverity statement, duration 2 applies to all assertions in the current fixture, etc.

For example, to make this version of vbUnit3 behave like vbUnit2, include the following statement in the first test method:

'vbUnit2 Compatibility

Public Sub TestVBUnit2Compatibility()
    m_assert.SetSeverity vbUnit_Assertion, vbUnit_AllTests 
   
   'now there can be several failed assertions in a single method
   ' for all tests in this run
   ... 
End Sub

Conversely, to speed up development cycles, make new fixtures the first fixture in your suite and set the severity to high. This will stop testing upon encountering the first problem, and you don't have to run the whole suite every time. This way of testing is very fast during development, but you can still be sure to run all tests for the complete project when the new fixture passes everything.

'Fastest development: Stop testing at first error or failure

'in the TestSuite:
Private Function ISuite_Suite() As ITest
Dim suite As New TestSuite
  suite.SuiteName = TypeName(Me)
    
  suite.AddFixture New NewestFixture
  suite.AddFixture New OlderFixture1
  suite.AddFixture New OlderFixture2
  
  Set ISuite_Suite = suite
End Function


'in the newest fixture:
Private Sub IFixture_Setup(assert As IAssert)
  Set m_assert = assert
  m_assert.SetSeverity vbUnit_AllTests, vbUnit_AllTests
End Sub

top

Local Severity Levels

The assertion methods Verify, LongsEqual, DoublesEqual, StringsEqual, and VariantsEqual have an aditional optional parameter that can override the current severity setting for this assertion. For example:

' stop all tests if the resource is invalid
m_assert.Verify m_resource.IsValid, "testResource", vbUnit_AllTests

This is basically a shorthand for setting the severity with a separate call to m_assert.SetSeverity.

top

Stopping tests

In addition to the security levels, IAssert also offers the method

Sub IAssert.StopTesting(scope As vbUnitScope)

This will break execution to the end of the required scope. For example,

m_assert.StopTesting vbUnit_Fixture

will exit the current fixture. Execution will continue at the next fixture.
In most cases, this can be achieved by having assertions fail. The only difference of the StopTesting method is that it won't list any failures in the TestResult.

top

Test-Resource Management

Fixture Lifetime

In most cases, the initialization for each test should be done in IFixture_Setup, and the cleanup in IFixture_TearDown. However, if a test fixture aquires very expensive resources such as database connections that take a long time to create, it may be desirable to do this only once for a fixture, or even just once for the complete test run.

A fixture may implement the optional interface IFixtureFrame, which defines the methods Create and Destroy. These methods will be called before and after ALL TestMethods in a given fixture have been called.

Class_Initialize and Class_Terminate are not good places to handle resource management, because these methods are already called when the fixture is added to a suite:

suite.AddFixture New TestVBUnit ' Class_Initialize for TestVBUnit is called here!

After the Fixture has been registered, it is destroyed again. Later, when the test is run, a second instance of the fixture class is created. Therefore, Class_Initialize should not acquire any expensive resources!

Here is the sequence of calls made to a fixture during its lifetime:

Class_Initialize ' suite.AddFixture
Class_Terminate

Class_Initialize                            ' run the actual Tests

    [ IFixtureFrame_Create ]        ' only called if class implements IFixtureFrame

        IFixture_Setup
                TestA
        IFixture_TearDown

        IFixture_Setup
                TestB
        IFixture_TearDown

        IFixture_Setup
                TestC
        IFixture_TearDown

    [ IFixtureFrame_Destroy ]      ' only called if class implements IFixtureFrame

Class_Terminate

Click here to find out how to do initialization and cleanup before and after running several suites.

top

Error handling for Create, Setup, TearDown, and Destroy

Each of those methods can throw failed assertions or errors, and the corresponding error or failure will be added to the TestResult. An error or failure in Setup or Teardown will stop all further tests in this fixture. In other words, the severity level (see below) for Create, Setup, Teardown, and Destroy is "vbUnit_Fixture". The following table lists the methods that are being called after an error or failure in each respective fixture method:

Error or Failure in:	Further methods called:
Class_Initialize	none (VB won't call Class_Terminate if there is an error in Class_Initialize)
Create	Destroy, Class_Terminate (no tests will be run at all)
Setup	Teardown, Destroy, Class_Terminate
(run TestMethod)	Teardown, Destroy, Class_Terminate
TearDown	Destroy, Class_Terminate
Destroy	Class_Terminate

top

Loading external files

The vbUnit3 TestRunner will set the current directory to the location of the compiled TestDll. All relative file references in your Test Project should be relative to that location. This works both in debug and in compiled mode, but obviously the TestDll needs to have been compiled at least once to indicate the correct directory.
Using relative instead of absolute file references is highly recommended, since it will allow you to move your test folders to a different location or machine much more easily. For example, the vbUnit3 folder can be installed anywhere without breaking the tests, since they only use relative paths.
New for version 3.05.00: If the compiled TestDll is not registered or not present, vbUnit will take the path of the target executable defined in the project settings. This is the file that VB suggests when you compile the project.

top

Manual fixtures (without reflections)

In some cases it may be desired to run tests without the use of reflections. This will allow to make the test fixture a private class of a project (Instancing = private), and it will also allow tests to be run on machines where Visual Studio is not installed. Manual fixtures use the interface IManualFixture instead of IFixture.

In addition to Setup and TearDown, IManualFixture has a third method that explicitly binds the test methods. Otherwise, it can be used exactly like the IFixture interface:

Private Sub IManualFixture_GetTestNames(tests As ITestNames)
    tests.Add "TestA"
    tests.Add "TestB"
    tests.Add "TestC"
End Sub

top