Mutation Analysis Research Project

For Software Testing and Reliability, my group decided to tackle what turned out to be a pretty difficult project. Our was titled “Comparing Statement Coverage and Number of Test Cases as Measures of a Test Suite’s Effectiveness in Finding Seeded Program Faults”. Our hypotheses was

We found some well-known open source python programs that had comprehensive test suites, that could be run with Python’s unittest module. My role was to find the programs, write the mutator, integrate our programs and collate the results.


All written in Python, used for statement coverage.


We were happy with the final results and some of the trend lines.


Mutation testing (seeding faults in a program and testing whether the test cases pick them up) is challenging. In Python, there were a couple of packages to do mutation testing, but they were quite fragile and didn’t quite suit our needs. We ended up writing a script to do the mutations for us, but due to time restrictions, we ended up writing quite an inefficient program that increased the time it took to test and collect results.

My CPU took a bit of a hit running the tests

They also took so long to run I had to do some “mobile computing”…

This also happened a few minutes after we submitted, right before the deadline. Lucky! Don’t leave things too late…