previous next

Step 5 - Unit Testing is not magic

In the last step we added a new class that implements the IStringBuffer interface with a different internal algorithm. By running the code in the debugger, we located and eliminated a bug that caused a run-time error. At the end of step 4, the InsertStringBuffer class passed the same unit test that we used for testing ConcatStringBuffer.
That means that as far as we can tell, the InsertStringBuffer class behaves exactly like the ConcatStringBuffer class.

One word of caution, though: Your confidence about the quality of your code should never be higher than the quality of your tests. The unit test that we have defined for testing the IStringBuffer interface only creates a string with 26 characters. It is possible that a StringBuffer implementation could handle that many characters, but not more. Right now, we can only be sure that InsertStringBuffer works correctly for up to 26 characters.

There are two ways to fix that: The Brute-Force approach, and The Smart Way. We could just write another test that creates strings of the maximum length that we expect to deal with, and this would probably even be a sensible thing to do in this case. However, in many real-world situtations it is completely impossible to exhaust the range of all possible values that your code will see. At the very least, it would be incredibly tedious and require a lot of time to write all those tests.

Code Coverage

Hence, we must be smart about what we test. Let's try to create a test that will tell us something new. One thing to look for is code coverage. Does our unit test call every line of code in the class to be tested? Maybe not. Look at the "Class_Initialize" method of InsertStringBuffer.cls:

'InsertStringBuffer.Class_Initialize

Private Sub Class_Initialize()
Const InitialSize As Long = 32

  m_buffer = String$(InitialSize, " ")
  m_allocated = InitialSize
  m_used = 0
End Sub

The initial size of the internal buffer is set to 32. However, we are only testing for 26 characters. This means that the "Grow" method may actually never get called. Ideally, our unit tests should cause the internal buffer to be grown several times, so that we can be sure that the algorithm for growing the buffer is working. If the buffer starts with 32 characters, it is doubled once after 64 characters, and twice after 128 characters. So let's write a unit test that adds a little more than 128 characters. Add the following piece of code to TestStringBuffer.cls:

'TestStringBuffer.TestFillLargeBuffer

Private Sub FillBufferBeyond128Chars()
Dim l As Long
Const ExpectedString As String = _
        "The initial size of the internal buffer is set to 32." _ 
        & vbCrLf & _
        "However, we are only testing for 26 characters. This " _
        & vbCrLf & _
        "means that the ""Grow"" method may actually " _ 
        &"never get called."
        
  m_assert.Verify Len(ExpectedString) > 128, "checkSourceString"
  
  For l = 1 To Len(ExpectedString)
    m_stringBuffer.Append Mid$(ExpectedString, l, 1)
  Next l
  
  m_assert.LongsEqual Len(ExpectedString), m_stringBuffer.Length, _
                      "checkLength2"
  m_assert.StringsEqual ExpectedString, m_stringBuffer.Result, _
                        "checkResult2"
End Sub

Public Sub TestFillLargeBufferConcat()
  Set m_stringBuffer = New ConcatStringBuffer
  FillBufferBeyond128Chars
End Sub
  
Public Sub TestFillLargeBufferInsert()
  Set m_stringBuffer = New InsertStringBuffer
  FillBufferBeyond128Chars
End Sub

Just for fun I have also included ConcatStringBuffer in the test, to see if it can handle this task as well. Fair is fair. Let's see the result of this test:


Figure 11: Filling a large buffer breaks InsertStringBuffer

This result means that 4 tests were executed (TestConcatenatingStringBuffer, TestInsertStringBuffer, TestFillLargeBufferConcat, and TestFillLargeBufferInsert).
Out of those, only one test failed, the last one. This means that ConcatStringBuffer passed this last test just fine, but InsertStringBuffer could not handle the long string. The result string seems to have the correct length, since the assertion m_assert.LongsEqual Len(ExpectedString), m_stringBuffer.Length, "checkLength2" didn't fire. But the beginning of the result string seems to contain only spaces. If you copy the result string into an editor that can count characters, you can quickly realize that exactly the first 128 characters of the broken result string are spaces. The last time the string doubled was from 128 to 256 characters. It seems that all characters added before the last reallocation were replaced with spaces.

Let's examine the "Grow" method of InsertStringBuffer:

'InsertStringBuffer.Grow (broken)

'double the allocated buffer
Private Sub Grow()
  m_buffer = String$(2 * Len(m_buffer), " ")
  m_allocated = Len(m_buffer)
End Sub

Here seems to be the problem. The first line of this method simply allocates a buffer filled with spaces without preserving the old content. That explains the strange TestResult that we got. Hence, change this method into the following:

'InsertStringBuffer.Grow (fixed)

'double the allocated buffer
Private Sub Grow()
  m_buffer = m_buffer & String$(Len(m_buffer), " ")
  m_allocated = Len(m_buffer)
End Sub

Run the tests again. This time all four tests should pass. 

Coverage of different situations

Does this mean that InsertStringBuffer is absolutely bulletproof now? Probably not. So far, we have only tested appending one character at a time. What would happen if we appended a string that is large enough to exceed the space gained by a single growth operation of the InsertStringBuffer? 

Let's find out:
The internal buffer of InsertStringBuffer starts with 32 characters. Since the internal buffer is doubled every time it grows, the first growth operation will increase its length to 64 characters. So let's try to add more than 64 characters at a time. Add the following code to TestStringBuffer.cls:

'TestStringBuffer.TestAppendLargeString

Private Sub AppendLargeString()
Const AppendString As String = _
"This test sentence has a length that clearly exceeds 64 characters."
  m_assert.LongsEqual 67, Len(AppendString), "checkAppendString"
  
  m_stringBuffer.Append AppendString
  
  m_assert.LongsEqual Len(AppendString), m_stringBuffer.length, _
                      "checkLength3"
  m_assert.StringsEqual AppendString, m_stringBuffer.Result, _
                        "checkResult3"
End Sub

Public Sub TestAppendLargeStringConcat()
  Set m_stringBuffer = New ConcatStringBuffer
  AppendLargeString
End Sub

Public Sub TestAppendLargeStringInsert()
  Set m_stringBuffer = New InsertStringBuffer
  AppendLargeString
End Sub

This time you should get:


Figure 12: Appending a large string also doesn't work yet

Our TestSuite now already contains 6 tests, and only one of them failed. Again, this means that ConcatStringBuffer passed the new test, and InsertStringBuffer didn't. Interestingly enough, InsertStringBuffer once again passed the first assertion m_assert.LongsEqual Len(AppendString), m_stringBuffer.length, "checkLength3". That means that it claims to have the correct expected length of 67 characters. However, checking the actual StringBuffer content reveals that the new string only contains 64 characters, and that the last 3 characters are missing. 

Obviously, adding a string that is longer than the amount of space gained by a single growth operation causes some problems. So let's fix this. Change the Append method of InsertStringBuffer.cls into the following:

'InsertStringBuffer.Append

Private Sub IStringBuffer_Append(str As String)
Dim length As Long
Dim requiredSize As Long

  'make sure the buffer is large enough and grow if necessary
  length = Len(str)
  requiredSize = m_used + length
  Do While requiredSize > m_allocated
    Grow
  Loop

  'insert new string into the buffer
  Mid$(m_buffer, m_used + 1) = str
  m_used = m_used + length
End Sub

Now run the tests again. This time the result should be:

  OK (6 Tests, 16 Assertions)  

At this point we have exercised every line of code of the InsertStringBuffer implementation, and we have tested all the different kinds of situations that this class may encounter. Except for pathological cases where the host computer runs out of memory, we can now be reasonably sure that InsertStringBuffer will provide the same logical functionality as the ConcatStringBuffer algorithm.

If you are really paranoid, you could add another test that creates a huge string composed of out of random segments with wildly different lengths, but it is unlikely that this would tell you something new. However, maybe we still have overlooked something in our discussion of the InsertStringBuffer implementation, or maybe there is a bug in VB, or in Windows, that only becomes visible under very specific conditions, so it would certainly not be unreasonable to write another test for the kind of input values that you expect in your application. But there is always a trade-off between the time spent on testing, and the time spent on development. Since the code in this example is fairly easy to analyze and understand, I would conclude that logically we have completely tested it now. Therefore, I suggest that we leave it as it is and move on. If any unexpected problems should occur later, we can still write more specific tests to pinpoint them.

 

Summary

This step has demonstrated that unit testing is not magic, and that bad tests can lead to a false sense of confidence. Writing good tests takes skill and deliberation, and that only comes with practice. Hence we went through so much detail to stress a few important points about unit testing:

  • you can only be sure that your code does something correctly when you have tested exactly that
  • knowing something about the implementation details of a class may allow you to write much smarter tests
  • a single test that fails unexpectedly may be worth more than 10 mechanically written exhaustive tests that pass
  • your tests should exercise every line of code of the class to be tested
  • your tests should focus on creating all the different kinds of situations that may arise, rather than stressing only a single kind of situation
  • writing tests helps you to understand the problem more clearly

 

previous next