Regression Testing -- A Valuable Software Testing Procedure

When you change the program, regression testing tests for things that should not change. If you automate your testing process, you can easily catch bugs that are newly introduced.

Download a Freeware Version


Regression Testing - What is it?
When Should You Run Regression Tests?
How To Run A Regression Test
Automate Regression Testing

Regression Testing - What is it?

Regression testing is a "management by exception" technique.

Regression testing tests for changes. Typically a regression test compares new output from a program against old output run using the same data. Unless, the reason for changing the program is to alter the output, the results should be identical.

Lets say the program was to add two numbers together.  In a previous test the results for adding 1 to 2 looked as follows:

Number One: 1
Number Two: 2
Total: 3

Lets say we modified this program to be a 32 bit program versus a 16 bit program. Lets also assume that a bug was introduced where number one was added to itself instead of the second number and the results look as follows:

Number One: 1
Number Two: 1
Total: 2

At first glance, it appears obvious that a simple review of the output will catch the mistake. What if there are 20 pages of computations? Would it be caught?

  1. Mistakes like this do happen.
  2. There is no guarantee that they will be caught by the right person and at the lowest cost.

In the above example, a regression test would catch the problem.

Back to Contents

When Should You Run Regression Tests?

The more frequently the better!

  • Anytime you add new features.
  • Anytime you tweak the system to improve performance.
  • Anytime you add a new component or third party control to the system. The same holds true for updating an old control or dll file.
  • Regression testing can be used to easily see if different operating systems or different versions of operating systems impact the way the system computes the results.
  • Whenever program maintenance is done to an existing system.
  • At each new phase of a new system.
  • Anytime you think someone could have inadvertently introduced (or deleted) some code.
  • It can be used to see if a bug in an old program has been cleared up.
  • Since this regression testing works with text files, this software testing measure can also be used with systems that generate Web pages.
  • You are a contract programmer, and you want to make sure that someone has not modified your work. Do you want to get blamed for other programmers' bugs?

Back to Contents

How To Run A Regression Test

  1. Using the previous program version that you are confident that the output is correct, run a test set of data to generate output in the form of a text file. This is what we call the Control Output File.
  2. With the newer version of the program, run the same test set of data that was used originally with the old version to generate output in the form of a text file. This file is what we call the New Output File.
  3. Compare the Control Output File to the New Output File. If the Control Output File was created with a properly functioning version of the program, both files should be the same. If they are not, you need to determine why they are different.
  4. Repeat steps 1 through 3 with for all scenarios that you want to test.

Back to Contents

Automate Regression Testing

If not automated, repeating the procedure to run the set of regression tests for each change can be time consuming, boring and subject to error. The trick is to automate the process. You can do things to automate regression testing as follows:

  • For each item you want to test for, you only need to create a Control Output File once, unless the new version requires a modification in the output.
  • Design the program being tested to take a command line that includes what the input file is, what the output file should be and something to indicate that the program start and exit without a user's intervention.
  • Use a regression testing utility like Denver Tax Software's Match2 that can run from a script or batch file.
  • Create a script or batch file that automatically runs the new version of the program and the regression testing utility for each test scenario.
  • Once the upfront work has been done to create the scripts or batch file, the regression testing can be scheduled to run at night, lunch or coffee breaks.
  • Check to see if an error log has been created.

Back to Contents


Denver Tax Software will soon be beta testing its Match2tm Automated Regression Tester. Send us email if you would like a copy of the beta version.
 Denver Tax Software, Inc.
PO Box 5308
Denver, CO 80112



Denver Tax Software, Inc.
PO Box 632285, Littleton, CO 80163-2285
Voice: 1-800-326-6686     Fax: 1-888-326-6686

Copyrightę 1997-2017, Denver Tax Software, Inc.