Running automated tests is a good thing for software quality assurance. Now you have to understand the results of the current test run. You could also be interested to see how your tests results evolve over time. To achieve these goals, you need a tool to produce meaningful tests reports and there are some open source tools to help you do this.
This article lists the main open source test reporting tools available today. Use our contact form to make us aware of some open source test reporting tools that we should add to this list.
Here are some features that you could find in open source test reporting tools:
- Managing data of one or many languages and test automation frameworks
- Creation of test reports in different formats: HTML, PDF, …
- Report distribution via e-mail
- Textual and graphical representation of results of a single test run
- Timelines graphs for presenting the evolution of results for different test runs
- Integration in the continuous integration / continuous delivery (CI/CD) process
This section lists tools that works with multiple programming languages and test automation frameworks.
ExtentReports / Klov
ExtentReports is an open source library that allows creating interactive and detailed reports for your tests. You can add events, screenshots, tags, devices, authors or any other relevant information you decide is important to create descriptive reports that you can totally control. This framework works with Java and C# languages and the Cucumber, SpecFlow and TestNG testing tools. ExtentReports is associated to Klov, a tool that allows tracking each of your launches across all your projects. You can view and analyze how your test perform overtime, how each execution category has compares in detail, track exceptions, create topics for discussion or reminders and search entities from a host of options. This open source framework has also a commercial version.
ReportPortal is an open source reporting service, that provides increased capabilities to speed up tests results analysis and reporting through the use of built-in analytic features. ReportPortal seamlessly integrates with mainstream CI/CD and project management platforms such as Jenkins, Jira and many functional and unit testing frameworks like Cucumber, JBehave, SpecFlow or Jasmine. Test case execution results are stored following the same structure you have in your reporting suites and test plan. The test cases are shown together with all related data in one place, right where you need it: logs, screenshots, binary data. The execution pipeline of certain test cases are also available for you, so one can see previous execution results in one click. ReportPortal also gives you the ability to collaboratively analyze the test automation results. Particular test cases can be associated with a product bug, an automation issue, a system issue or can be submitted as an issue ticket directly from the execution result. With each execution, ReportPortal automatically figures out the root cause of a fail. As a result of this analysis, ReportPortal is marking a test result with a flag. Engineers will be alerted about this issue to provide further analysis: if it has been resolved already or which test results require actual human analysis.
This section lists tools that works with a limited number of programming languages and test automation frameworks.
CI::Reporter is an add-on to Ruby testing frameworks like Test::Unit or RSpec that allows you to generate XML reports of your test runs. The resulting files can be read by a continuous integration system that understands Ant’s JUnit report XML format, thus allowing your CI system to track test/spec successes and failures. CI::Reporter works with projects that use standard Rake tasks for running tests. In this fashion, it hooks into testing frameworks using environment variables recognized by these custom tasks to inject the CI::Reporter code into the test run.
Cluecumber Report Maven Plugin
Cluecumber Report Maven Plugin is an open source maven plugin for clear and concise Cucumber BDD test reporting. This plugin creates aggregated test reports from Cucumber compatible JSON files that are generated by Cucumber BDD, Karate and other frameworks.
karma-htmlfile-reporter is a karma plugin for exporting unit test results as styled HTML file. This is a plugin for the Karma Test Runner. By adding this reporter to your karma configuration, unit test results will be exported as a styled HTML file. For each test browser, a separate table is generated.
ReportBuilder is a Ruby gem that merges Cucumber JSON reports and builds mobile-friendly HTML Test Report, JSON report and retry file.
ReportNG is a simple HTML open source reporting plug-in for the TestNG unit-testing framework. It is intended as a replacement for the default TestNG HTML report. The default report is comprehensive but is not so easy to understand at-a-glance. ReportNG provides a simple, color-coded view of the test results. ReportNG generates 100% valid XHTML 1.0 files. The output can be customized by over-riding the default stylesheet with your own CSS file.
Spock Reports Extension is an open source global extension for Spock to create test (or, in Spock terms, Specifications) reports. By default, the report creator generates an HTML report for each Specification, as well as a summary of all Specifications that have been run (index.html). If you prefer to have your own template to generate reports from, you can use the TemplateReportCreator. This allows you to generate reports in any text format.
XCTestHTMLReport is an open source Xcode-like HTML report for Unit and UI Tests. It supports parallel testing and attachments.