SDSU CS 580 Client-Server Programming
Fall Semester, 2002
Testing
Previous    Lecture Notes Index    Next    
© 2002, All Rights Reserved, SDSU & Roger Whitney
San Diego State University -- This page last updated 29-Sep-02

Contents of Doc 11, Testing



References

JUnit Cookbook http://junit.sourceforge.net/doc/cookbook/cookbook.htm

JUnit Test Infected: Programmers Love Writing Tests http://junit.sourceforge.net/doc/testinfected/testing.htm

JUnit Javadoc: http://www.junit.org/junit/javadoc/3.8/index.htm

Brian Marick’s Testing Web Site: http://www.testing.com/

Testing for Programmers, Brian Marick, Available at: http://www.testing.com/writings.html


Doc 11, Testing Slide # 2
Reading Assignment

Test Infected - Programmers Love Writing Tests, http://junit.sourceforge.net/doc/testinfected/testing.htm

JUnit FAQ, http://junit.sourceforge.net/doc/faq/faq.htm

Programming with Assertions, http://java.sun.com/j2se/1.4/docs/guide/lang/assert.html

Lecture Source Code

Java – CVS module testExamples

Smalltalk – Store package testExamples


Doc 11, Testing Slide # 3

Testing


Johnson's Law


If it is not tested it does not work



Types of tests

Tests individual code segments
Test functionality of an application


Doc 11, Testing Slide # 4

Why Unit Testing


If it is not tested it does not work


The more time between coding and testing
  • More effort is needed to write tests
  • More effort is needed to find bugs
  • Fewer bugs are found
  • Time is wasted working with buggy code
  • Development time increases
  • Quality decreases

Without unit tests
  • Code integration is a nightmare
  • Changing code is a nightmare

Doc 11, Testing Slide # 5
Why Automated Tests?

What wrong with:




Doc 11, Testing Slide # 6
Repeatability & Scalability


Need testing methods that:





Practices that work in a school project may not be usable in industry

Standard industry practices may seem overkill in a school project

Work on building good habits and skills


Doc 11, Testing Slide # 7
We have a QA Team, so why should I write tests?


How long does it take QA to test your code?

How much time does your team spend working around bugs before QA tests?

How easy is it to find & correct the errors after QA finds them?


Most programmers have an informal testing process

With a little more work you can develop a useful test suite



Doc 11, Testing Slide # 8

When to Write Unit Tests


First write the tests

Then write the code to be tested



Writing tests first saves time

  • Removes temptation to skip tests


Doc 11, Testing Slide # 9

SUnit & JUnit


Free frameworks for Unit testing

SUnit originally written by Kent Beck 1994


JUnit written by Kent Beck & Erich Gamma


Already installed in JDK 1.2 on rohan and moria


Ports are available in


.NET
Ada
AppleScript
C
C#
C++
Curl
Delphi
Eiffel
Eiffel
Flash
Forte 4GL
Gemstone/S
Haskell
HTML
Jade
LISP
Objective-C
Oracle
Palm
Perl
Php
PowerBuilder
Python
Ruby
Scheme
Smalltalk
Visual Basic
XML
XSLT



See http://www.xprogramming.com/software.htm to download ports


Doc 11, Testing Slide # 10

How to Use SUnit


1. Make test class a subclass of TestCase


2. Make test methods
The framework treats methods starting with 'test' as test methods

3. Run the tests

You can run the test using TestRunner
      TestRunner open
Use Browser SUnit Extensions
Load parcel/RBSUnitExtensions.pcl to run the tests from the browser

Doc 11, Testing Slide # 11
Broswer SUnit Extensions



Doc 11, Testing Slide # 12
Sample TestClass

Smalltalk.CS535 defineClass: #TestCounter
   superclass: #{XProgramming.SUnit.TestCase}
   indexedType: #none
   private: false
   instanceVariableNames: 'counter '
   classInstanceVariableNames: ''
   imports: ''
   category: 'Course-Examples'

CS535.TestCounter methodsFor: 'testing'

setUp
   counter := Counter new.

tearDown
   counter := nil.

testDecrease
   counter decrease.
   self assert: counter count = -1.

testDecreaseWithShould
   "Just an example to show should: syntax"
   counter decrease.
   self should: [counter count = -1].

testIncrease
   self deny: counter isNil.
   counter increase.
   self assert: counter count = 1.

testZeroDivide
   "Just an example to show should:raise: syntax"
   self 
      should: [1/0]
      raise: ZeroDivide.
   
   self 
      shouldnt: [1/2]
      raise: ZeroDivide


Doc 11, Testing Slide # 13

TestCase methods of interest


Methods to assert conditions:

   assert: aBooleanExpression
   deny: aBooleanExpression
   should: [aBooleanExpression]
   should: [aBooleanExpression] raise: AnExceptionClass
   shouldnt: [aBooleanExpression]
   shouldnt: [aBooleanExpression] raise: AnExceptionClass
   signalFailure: aString
    
setUp
Called before running each test method in the class.
Used to:
Open files
Open database connections
Create objects needed for test methods
tearDown
Called after running each test method in the class.

Used to:
Close files
Close database connections
Nil out references to objects


Doc 11, Testing Slide # 14

Using JUnit

Example

Goal: Implement a Stack containing integers.

Tests:

Subclass junit.framework.TestCase
Methods starting with 'test" are run by TestRunner
First tests for the constructors:

public class  TestStack extends TestCase {
   //required constructor
   public TestStack(String name) {
      super(name);
   }
   
   public void testDefaultConstructor() {
      Stack test = new Stack();
      assertTrue("Default constructor", test.isEmpty() );
   }
      
   public void testSizeConstructor() {
      Stack test = new Stack(5);
      assertTrue( test.isEmpty() );
   }
   
}


Doc 11, Testing Slide # 15
First part of the Stack

package example;
public class Stack  {
   int[] elements;
   int topElement = -1;
   
   public Stack() {
    this(10); 
   }
   public Stack(int size) { 
      elements = new int[size]; 
   }
   
   public boolean isEmpty() {
      return topElement == -1;
   }
}

Doc 11, Testing Slide # 16

Running JUnit


JUnit has three interfaces

Fastest to run

Can reload class files so you can
Run TestRunner once
Recompile program until it passes tests



Doc 11, Testing Slide # 17
Starting Swingui TestRunner

Make sure your classpath includes the code to tested

On Rohan use:

java  junit.swingui.TestRunner
You get a window similar to that on the next page


Enter the full name of the test class

Click on the Run button

If there are errors/failures select one

You will see a stack trace of the error

The “...” button will search for all test classes in your classpath


Doc 11, Testing Slide # 18
Swing version of JUnit TestRunner




Doc 11, Testing Slide # 19
Running the textui TestRunner

Sample Program using main

public class Testing {
   public static void main (String args[]) {
      junit.textui.TestRunner.run( example.TestStack.class);
   }
}

Output

..
Time: 0.067
OK (2 tests)
java has exited with status 0.


Doc 11, Testing Slide # 20

assert Methods


assertTrue()
assertFalse()
assertEquals()
assertNotEquals()
assertSame()
assertNotSame()
assertNull()
assertNotNull()
fail()

For a complete list of the assert methods & arguments see

http://www.junit.org/junit/javadoc/3.8/index.htm/allclasses-frame.html/junit/junit/framework/Assert.html/Assert.html


Doc 11, Testing Slide # 21
JUnit, Java 1.4 & assert

JUnit had a method called assert()

Java 1.4 makes assert a reserved word

JUnit starting 3.7 replaces assert() with assertTrue()

Use JUnit 3.7 or later with JDK 1.4


To use JDK 1.4 asserts:


   java -source 1.4 programFile.java


   java -ea programFile



Doc 11, Testing Slide # 22

Testing the Tests


If can be useful to modify the code to break the tests

package example;
public class Stack  {
   int[] elements;
   int topElement = -1;
   
   etc.
   
   public boolean isEmpty() {
      return topElement == 1;
   }
}

Doc 11, Testing Slide # 23
Result of running Textui.TestRunner
.F.F
Time: 0.113
There were 2 failures:
1) testDefaultConstructor(example.TestStack)junit.framework.AssertionFailedError: Default constructor
at example.TestStack.testDefaultConstructor(TestStack.java:22)
at Testing.main(Testing.java:14)
2) testSizeConstructor(example.TestStack)junit.framework.AssertionFailedError
at example.TestStack.testSizeConstructor(TestStack.java:27)
at Testing.main(Testing.java:14)

FAILURES!!!
Tests run: 2, Failures: 2, Errors: 0


java has exited with status 0.

Doc 11, Testing Slide # 24
Why Test the Tests?

One company had an automatic build and test cycle that ran at night. The daily build was created and all the tests were run at night. The test results were available first thing in the morning. One night the build process crashed, so the daily build was not made. Hence there was no code to test. Still 70% of the tests passed. If they had tested their tests, they would have discovered immediately that their tests were broken.



Doc 11, Testing Slide # 25

Test Fixtures


Before each test setUp() is run

After each test tearDown() is run

package example;
import junit.framework.TestCase;
public class  StackTest extends TestCase {
   Stack test;
   
   public StackTest(String name) {
      super(name);
   }
   
   public void setUp() {
      test = new Stack(5);
      for (int k = 1; k <=5;k++)
         test.push( k);   
   }
   public void testPushPop() {
      for (int k = 5; k >= 1; k--)
         assertEquals( "Pop fail on element " + k,  test.pop() , k);
   }
}


Doc 11, Testing Slide # 26

Suites – Multiple Test Classes


Multiple test classes can be run at the same time

Add Queue & TestQueue to Stack classes

package example;
import junit.framework.TestCase;
public class TestQueue extends TestCase{
   public TestQueue ( String name){
      super(name);
   }
   public void testConstructor() {
      Queue test = new Queue();
      assert( test.isEmpty());
   }
}

package example;
import java.util.Vector;
public class  Queue{
   Vector elements = new Vector();
   public boolean isEmpty() {
      return elements.isEmpty();
   }
}


Doc 11, Testing Slide # 27
Using a Suite to Run Multiple Test Classes

Running AllTests in TestRunner runs the test in

StackTest
QueueTest

package example;
import junit.framework.TestSuite;
import junit.textui.TestRunner;
public class AllTests  {
   static public void main(String[] args) {
       TestRunner.run( example.AllTests.suite());;
   }
   static public TestSuite suite() {
         TestSuite suite= new TestSuite();
      Try {
          suite.addTest(new TestSuite(StackTest.class));
          suite.addTest(new TestSuite(QueueTest.class));
       } catch (Exception e){
       }
       return suite;
   }
}


Doc 11, Testing Slide # 28
Using Main

We can use main to run the test via textui.TestRunner

The command:

   java example.AllTests
will run all the tests in StackTest & QueueTest

package example;
import junit.framework.TestSuite;
import junit.textui.TestRunner;
public class AllTests 
   {
   static public void main(String[] args) 
      {
      TestRunner.run(AllTests.class);
      }
      
   static public TestSuite suite()
      {
      same as last page
      }
   }

Doc 11, Testing Slide # 29

How to Test Exceptions


At times you may wish to test input to methods that will cause an exception to be thrown

Here is an example of a test that

 
Example is from the JUnit FAQ


public void testIndexOutOfBoundsException() {
      
    ArrayList list = new ArrayList(10);
      
    try {
   
        Object o = list.get(11);
   
        fail("Should raise an IndexOutOfBoundsException");
   
    } catch (IndexOutOfBoundsException success) {}
}


Doc 11, Testing Slide # 30

Testing and Hidden Methods/State


Issues:




Doc 11, Testing Slide # 31
Testing Hidden Methods One Position Don't Do it

Pro:


The basic idea is to work smarter not harder. One cannot completely test each class, and one does not have infinite time two write tests one should write the most effective tests possible. Tests of the public interface of class will also test hidden methods of the class. Bugs in hidden methods that never affect the public methods are not a problem. Since the most important thing is that the pubic interface works correctly, concentrate your tests on the public interface.

Con:


My experience is that the more code I write without tests, the more time I spend on finding and correcting bugs. How many times have you spent hours (days?) tracking down a bug that turned out to a simple bug in some simple untested method, which would have been easy to test? The argument that one cannot test everything and must make effective use of ones testing time is correct. Given the differences in programmer skill level, programmer experience, etc. everyone has to work out their own solution to this. The XP solution is to try to test everything that could possible break. Since most students are not used to testing, you have to fight the habit of not testing and testing after you have completely finished coding. Given the current state of affairs in commercial software, the industry has a lot to learn about testing.

Doc 11, Testing Slide # 32

How to Test Hidden Methods Directly?

Method 1: Relax the protection level


In Java one can



Pro:



Con:


You should comment the method to inform the clients that the method is not to be used

Doc 11, Testing Slide # 33
How to Test Hidden Methods Directly?

Method 2: Use inner classes


import junit.framework.TestCase;
public class Foo {
    private int value;
    private void bar() {
        value = 10;
    }
    public static class FooTest extends TestCase {
        public FooTest(String name) {
            super(name );
        }
        public void testBar() {
            Foo a = new Foo();
            a.bar();
            assert( 10 == a.value );
        }
    }
}

Pro:

Con:


Doc 11, Testing Slide # 34
How to Test Hidden Methods Directly?

Method 3: Use reflection


Pro:

Con:

See http://www.eli.sdsu.edu/courses/fall98/cs596/notes/reflection/reflection.html for more information about reflection




Doc 11, Testing Slide # 35

What to Test


Everything that could possibly break


Test values

Inside valid range
Outside valid range
On the boundary between valid/invalid


GUIs are very hard to test

Keep GUI layer very thin
Unit test program behind the GUI, not the GUI


Doc 11, Testing Slide # 36
Common Things that Programs Handle Incorrectly

Adapted with permission from “A Short Catalog of Test Ideas” by Brian Marick, http://www.testing.com/writings.html


Strings

Test using empty String

Collections

Test using:

Numbers

Test using:


Doc 11, Testing Slide # 37

Testing Network Code


Writing automated tests for network code can be hard

Make the network code very thin

Write automated tests for the non-network code


Doc 11, Testing Slide # 38
Example – SimpleDateServer

The SimpleDateServer has two methods

Write tests for processRequestOn:

SimpleDateServer>>run
   | childSocket clientIOStream |
   [childSocket := serverSocket accept.
   clientIOStream := childSocket readAppendStream.
   clientIOStream lineEndTransparent.
   self processRequestOn: clientIOStream.] repeat

SimpleDateServer>>processRequestOn: anReadAppendStream 
   | clientRequest |
   clientRequest := anReadAppendStream through: Character cr.
   (clientRequest startsWith: 'date') 
      ifTrue: 
         [anReadAppendStream
            nextPutAll: Time dateAndTimeNow printString;
            commit].
   anReadAppendStream close


Doc 11, Testing Slide # 39
Issues in Writing test for processRequestOn:

processRequestOn: Input
Need to create an ExternalAppendStream for input

Would be easier to test if arguments were:

Server Output

Initial sever was written without thought to parsing output

It will be hard for tests to parse current output

So it will be hard for a client to parse server response

Conclusion:



Doc 11, Testing Slide # 40
Modified Server

SimpleDateServer>>run
   | childSocket |
   
   [childSocket := serverSocket accept.
   self 
      processRequest: childSocket readStream lineEndTransparent
      response: childSocket writeStream lineEndTransparent] 
         repeat

SimpleDateServer>>processRequest: aReadStream response: aWriteStream
   | clientRequest |
   clientRequest := aReadStream through: Character cr.
   (clientRequest startsWith: 'date') 
      ifTrue: 
         [aWriteStream
            nextPutAll: Date today printString;
            commit].
   (clientRequest startsWith: 'time') 
      ifTrue: 
         [aWriteStream
            nextPutAll: Time now printString;
            commit].
   aWriteStream close.


Doc 11, Testing Slide # 41
Some Tests

TestSimpleDateServer is subclass of TestCase

TestSimpleDateServer>>testDate
   | command response server serverDate serverDateString |
   command := 'date \' withCRs readStream.
   response := WriteStream on: String new.
   server := SimpleDateServer new.
   server processRequest: command response: response.
   serverDateString := response contents.
   serverDate :=Date readFrom: serverDateString readStream. 
   self assert: serverDate = Date today.

TestSimpleDateServer>>testTime
   | command response server serverTimeString serverTime |
   command := 'time \' withCRs readStream.
   response := WriteStream on: String new.
   server := SimpleDateServer new.
   server processRequest: command response: response.
   serverTimeString := response contents.
   serverTime :=Time readFrom: serverTimeString readStream. 
   self assert: (Time now asSeconds - serverTime asSeconds) < 2.
   
TestSimpleDateServer>>testBadInput
   | command response server |
   command := 'fooBar \' withCRs readStream.
   response := WriteStream on: String new.
   server := SimpleDateServer new.
   server processRequest: command response: response.
   self assert: response contents trimBlanks isEmpty

Doc 11, Testing Slide # 42
Lessons

Keep the network layer thin

Server design may need to be modified to make it testable


See
Test networked code the easy way, Nelson Minar
http://www.javaworld.com/javaworld/jw-07-2002/jw-0719-networkunittest.html

for another example

Copyright ©, All rights reserved.
2002 SDSU & Roger Whitney, 5500 Campanile Drive, San Diego, CA 92182-7700 USA.
OpenContent license defines the copyright on this document.

Previous    visitors since 29-Sep-02    Next