News
Every change to CPO runs 306 regression tests first
A compiler upgrade, a new user interface and a C++ modernisation are all planned. Before any of it ships, CPO's 306 test cases run automatically and every number is compared against a recorded baseline.
CPO has a lot of work planned: a compiler upgrade (to enable cross platform and multi-processors), a migration of the user interface, a modernisation of the C++ underneath and a lot of new functionality. Each one carries the a risk that a change meant to improve quality quietly alters a simulation result. That is not something we want to hear about from a customer.
So the test rig came first. It runs CPO's whole body of test cases and compares the results against a recorded baseline.
Not starting from scratch
None of this was new ground. When CPO2D was converted from 32-bit to 64-bit we built the machinery to check that the converted program still gave the same answers. The rig is that machinery extended: the same approach, widened to cover every example and shape data file, in 3D as well as 2D.
306 cases, 99 in 2D and 207 in 3D
The test inputs already exist in each CPO installation - 99 data files for CPO2D and 207 for CPO3D, 306 in total, each producing defined output files. Around them sits the rest of the rig: a runner that executes every case, a baseline holding the output each case is known to produce, and a comparison tool that reads a fresh run against that baseline and reports where the two disagree.
A clean environment for every run
Each case runs in its own environment, isolated from every other, and every environment is built from scratch at the start of the run. Nothing is shared between cases and nothing survives from the run before. Think of it as rebooting the universe from the apps point of view before every test case.
Comparison to about 0.01%
Floating-point output might not be bit-identical between runs (depending on the test and the stochastic nature of the calculations), so a straight file diff can report issues that are not failures. A variation of around 0.01% between runs is normal. The comparison is therefore value-by-value against a tolerance rather than byte-by-byte. Outside that tolerance, it is a regression.
Timings are recorded as well as numbers
Every run records how long each case took, against the same baseline. Code that returns identical results more slowly is still a regression. This class of test will help us ratchet up performce as we go.
Nothing ships on a run that is not clean
No compiler port, no interface migration and no C++ modernisation goes out on a failing run. Where a run disagrees with the baseline, we find out why before anything is released. Sometimes the change is wrong. Sometimes the baseline is, and then it gets updated deliberately and recorded, rather than quietly overwritten to make the run go green.
Running it on your own hardware
The suite is not only for us. Power users who want to validate a release against their own machines before upgrading should be able to do so and we intend for this work to be publicly available if proven useful.
Where this stands
The rig runs today. The cases, the baselines, the comparison and the timings are all in place, it already exists ahead of the work it is there to protect, rather than being assembled once something has already gone wrong. The compiler upgrade, the interface migration and the C++ modernisation are all still in front of it.
We'll be reporting on progress as we go, and on any interesting issues we encounter.