Skip to content

make plotter tests idempotent in cases where they run to completion

Sebastien Binet requested to merge test_golden into master

Created by: ctessum

This is in response to #252 (closed). Ideally, the golden images would only be edited in cases where the user requests it. However, in order for the example functions to use p.Save() (which is probably desirable to help people understand the example functions), they need to write directly to the file system every time. If we completely separate the test functions from the example functions, then we no longer have any assurance going forward that the example functions are current and produce the files in the 'testdata' directory.

As a compromise solution, I've changed checkPlot() so that it runs the function as before, which overwrites the golden image, but if the new image is different than the previous golden image, it writes out the previous golden image in it original location and writes out the new image with the "_errored" suffix.

A drawback of this is that if the test's don't run to completion---for example, if the hard drive runs out of space in the middle of the test---the golden images may not be reverted to their original versions. However, this problem would be apparent in a pull request comparison on github, so it is not too subtle.

Other options:

  1. Instead of using p.Save() in the example functions, have them write to a buffer, which could be compared to the golden image without overwriting it. I think this would make the examples substantially more confusing. We could also use a "hidden" save function and explain that a user would normally use p.Save(), but I put this in a previous version and it was not popular.
  2. Delete the example functions so that there are only test functions in the source code itself, and move the examples to a separate gallery. I'm skeptical of this because then the gallery would require separate maintenance to keep it current with the library, and i've seen cases in other libaries where the example functions don't run or they don't produce the advertised output, which is frustrating.

Merge request reports