Approval Testing is an approach to software testing that focuses on capturing and storing the behavior of your application, presenting the changes in behavior and allowing the software development team to react to them. There are some open source tools that will help you automate approval testing.
Approval testing can work both in unit testing or functional testing activities. The principle is simple: when you define your tests, you declare a reference result that could be for instance a bank account balance or an order summary screen. When you run your test, the tool will spot all the differences between the reference result and the actual result. You can either approve the difference, because you changed something in your code, ignore the difference as not meaningful or you can mark the test as failed. Approval testing tries to solve the issue faced when you try to test a feature that has as an output something more complex than a simple number, a case where you can use assertion-based testing. Approval testing works also fine in the situation where you maintain legacy code developed by somebody else. You want to be sure that single module modification has not negative side results elsewhere. Approval testing is also mentioned on the web under the names of “Golden Master Testing”, “Characterisation Testing” or “Snapshot Testing”.
The limitation of approval testing is that results have to be checked by a human, thus making it less suitable for the continuous testing/delivery approach where the whole application would be tested and build every night for instance. The fact is that you can also automate the “approve” and “ignore” operations.
Tools
Approval testing is a software testing technique that really works with some test automation support. You need a different test tool for the different context where you are operating, as you cannot necessarily test java code and HTML screens with the same tool. Many tools support approval testing. Many of them are grouped under the “Approvals” family. Here is a little list of open source tools that allows you to perform approval testing. Use the contact form to let us know of any tool that has not yet been included in this list.
* Approvals Tests
Approval Tests Library is designed to augment a unit test framework with approval testing capabilities. It is available for Java, C#, VB.Net, PHP, Ruby, Node.JS and Python.
Website: http://approvaltests.com/
* TextTest
TextTest is an open source tool for text-based functional testing. This means running a program in lots of different ways from the command line, and using the text output produced as a means of controlling the behavior of that application. TextTest works via comparing plain text logged by programs with a previous ‘gold standard’ version of that text. This is in contrast to most acceptance testing frameworks on offer today, which generally use some form of handwritten ‘assertions’ by the test writer that call into an application API. For Python and Java GUI, you can use the open source tool StoryText, which is especially designed to work with TextTest.
Website: http://texttest.sourceforge.net/
* Jest
Jest is an open JavaScript testing solution developed by Facebook that can be used by any React project. It is a very useful tool whenever you want to make sure your UI does not change unexpectedly. A typical snapshot test case for a mobile app renders a UI component, takes a screenshot, then compares it to a reference image stored alongside the test. The test will fail if the two images do not match: either the change is unexpected, or the screenshot needs to be updated to the new version of the UI component.
Website: https://jestjs.io/
* Recheck
Recheck is an open source software testing framework that allows replacing manual asserts and checking everything at once. It allows creating and maintaining checks for web and Java Swing with a semantic comparison of contents.
Website: https://github.com/retest/recheck
* Automated Screenshot Diff
Automated Screenshot Diff allows you and your team to extend test coverage up to the application layout, making sure you are aware of any layout changes whether you use manual or automated testing strategies. All you have to do is generate your system’s screenshots, then automated-screenshot-diff will scan your screenshots folder and calculate differences between your pre-production (stage) release and your production release.
Website: https://github.com/igorescobar/automated-screenshot-diff
* Depicted (dpxdt)
Make continuous deployment safe by comparing before and after web page screenshots for each release. Depicted (dpxdt) shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test. Here are the steps to making Depicted useful to you:
1. Establish a baseline release with an initial set of screenshots of your site.
2. Create a new release with a new set of screenshots of your new version.
3. Manually approve or reject each difference the tool finds.
4. Manually mark the new release as good or bad.
5. Repeat. Your approved release will become the baseline for the next one.
Website: https://github.com/bslatkin/dpxdt
* BackstopJS
BackstopJS is an open source tool that automates visual regression testing of your responsive web UI by comparing DOM screenshots over time. It supports screen rendering with Chrome-headless, Phantom and SlimerJS. You can add your own interactions with Puppeteer, ChromyJS and CasperJS scripting.
Website: https://garris.github.io/BackstopJS/
* Appraise
Appraise is an open source tool for visual approval testing that is still in its early development phase. It can help you review and approve changes to web pages, visual layouts and browser components quickly through visual inspection. Appraise allows automating acceptance or regression tests for visual look and feel in a visual language, rather than xUnit style code.
Website: https://github.com/AppraiseQA/appraise/
References
* Approval Testing: Agile Testing that Scales
* Approval Tests, Alternative View on Test Automation
* Approval Testing with TextTest
* Approval testing with Jasmine
* Approval Tests – Overview
* Characterization Testing
* Surviving Legacy Code with Golden Master and Sampling
* What is the “Golden Master” Technique?