For Software Testing and Reliability, my group decided to tackle what turned out to be a pretty difficult project. Our was titled “Comparing Statement Coverage and Number of Test Cases as Measures of a Test Suite’s Effectiveness in Finding Seeded Program Faults”. Our hypotheses was
We found some well-known open source python programs that had comprehensive
test suites, that could be run with Python’s unittest module. My role was
to find the programs, write the mutator, integrate our programs and collate the
results.
Tech
All written in Python, used coverage.py for statement coverage.
Results
We were happy with the final results and some of the trend lines.
Learning
Mutation testing (seeding faults in a program and testing whether the test cases pick them up) is challenging. In Python, there were a couple of packages to do mutation testing, but they were quite fragile and didn’t quite suit our needs. We ended up writing a script to do the mutations for us, but due to time restrictions, we ended up writing quite an inefficient program that increased the time it took to test and collect results.
My CPU took a bit of a hit running the tests
They also took so long to run I had to do some “mobile computing”…
This also happened a few minutes after we submitted, right before the deadline. Lucky! Don’t leave things too late…
