The creation of a TestOps infrastructure is a demanding task. TestOps (or test operations) refers to the discipline of managing the operational aspects of testing within the software delivery lifecycle. In this article, we focus on theoretical and technical aspects, and discuss some case studies to consider.
Author: Maksim Sarychau, Test Automation Solution Architect and QA Engineering Manager at Solvd
Theoretical aspects
In the first part of this article, we discuss test maintainability based on Object Oriented Programming (OOP) best practices, how to keep automation results transparent for management, and other theoretical aspects of TestOps infrastructure.
Reaching all platforms
Increasingly, customers demand a cross-platform solution. From Web, Mobile, AI, to gaming consoles, a wider array of clients means considering all platforms. In order to make your tests truly integrated, you can use an all-in-one open-source solution called Carina. If you prefer, you can use any framework you want, though using a more complex solution will save time and money.
Testing Maintainability
When it comes to test maintainability, you should adhere to the main practices and good projecting patterns that are based on object-oriented programming. Using page objects for functional testing is a trending practice. However, these tests are difficult to maintain; look outside of the ‘proof-of-concept’ aspect for a clearer overview.
Pro-tip: use an abstraction of UI for mobile automation with iOS and Android. In this way, you can reuse source code without having to make drastic changes to it.
Using page objects and UI components should be done as much as possible to maintain a clear structure of tests whilst maintaining a link to business logic. UI components should be gathered into differing groups: menus, controls, etc.
UI validations and REST calls allow for stable testing, though this methodology requires data parametrization using the following aspects:
- Parametrize tests from Excel and databases
- Utilize many tests and cover all domain areas
- Combine REST calls with UI validation for stability
Keeping automation results transparent
Managers prefer visual representations, so charts and linkable reports are necessary. Follow orthodox approaches, offering visibility across development/manual QA. Testing from a specific approach of test automation engineers, for example, is too one-sided to get legible results: visibility across teams is necessary. Give management what they want: results represented clearly. In this sense, clear visualization is the key to all teams understanding automation results.
Similarly, focus on fast automation results. Ideally, test automation is integrated into CI/CD process. Visibility is again necessary here, keep results simple and develop trust. Explain test procedures properly, as well as integration processes. Additionally, make small test suites running in small chunks for clarity. Developers prefer this approach, leading to an effective relationship between them and the QA department.
Ensure that manual QA can interact with results. Offer tools to check tests results: logs, artifacts, video or screenshots. If tests regularly fail, explain why: is it the testing environment? Are there issues with Selenium or Appium? Once result factors are communicated, manually checking tests can be done by QA engineers, saving time.
Visualizations and reports allow your QA management to track progress, velocity, and productivity, giving them a chance to offer their input on the process, potentially improving time management.
Full integration into the CI/CD process
Depending on preference, automation can be scheduled, or manual. Concerning integration into the CI process, allow your test automation to be trigged on a commit and pull request basis – leave the opportunity to start tests manually, as well as regular tests also.
With flexible configuration, the AUT environment/execution environment can be selected. Tests can be customized by choosing an environment, browser, or mobile device.
Software issues, such as in Selenium, can waste resources. For example, 50/300 tests failed, and rerunning the whole regression is futile. Choose open-source products that allow you to rerun only failed tests or their sequences to collate reports.
Recommended open-source tools:
- Selenium for web automation and Appium for mobile
- For test life-cycle control, use TestNG. It’s more versatile compared to competitors.
- Maven SureFire Plugin allows for running certain groups of tests under CI to parametrize them.
- Use Selenoid as a text execution environment: it allows you to run tests in a dockerized environment and separate Docker container, via browsers and mobile.
- For real-time or historical reports, use RabbitMQ, and Elastic Stack for arching logs and future analysis.
- Mobile automation can be run through an MCloud device farm. Connect these to a server through hubs and it can be used by manual and automation QA engineers.
- Open STF is an excellent open-source tool allowing you to control outside devices – swipe, type, click, install apps.
Tools and technologies for orchestrating infrastructure components, testing, and continuous integration and delivery are described in this passage. Ansible is used for automating deployment and configuration, while SeleniumGrid or AppiumGrid integrated with Docker containers are used for the testing infrastructure. Kubernetes can be used to administer the infrastructure if it is hosted on platforms like Amazon or Google Cloud. Jenkins is the preferred open-source CI/CD tool, and pipelines are used to configure and run tests after a build.
Practical approach
The second part of this article discusses the setup process and emphasizes the importance of certain elements for the system to function correctly.
In the technical part of the article, we focus on practical implementation. Our company has developed a set of tools over the past decade that work well for our projects and clients. We recommend using tools that work best for your specific needs, such as automation scripts and reporting tools for visualization. This approach has been tried and tested in production and can be replicated successfully.
Carina/Zebrunner Logging
Zebrunner is a Java-based web application built on the Spring Framework, used for aggregating test results from Carina through the Zebrunner Listener Agent. The results are displayed in a web interface that includes graphs and interactive dashboards. ZebrunnerLogAppender is a Log4j Appender that publishes logging events to RabbitMQ, which can be viewed in real-time through the Zebrunner web reporting tool.
To access historical test logs, the Elastic Stack (Elasticsearch and Logstash) is used to store the data. A Logstash pipeline is set up to receive input from RabbitMQ and record the data with the corresponding Test Run ID and Test ID in Elasticsearch. This allows users to retrieve relevant test logs by forming an HTTP request in Zebrunner.
The process: to review test logs later, a pipeline is established in LogStash that receives input from RabbitMQ. This data in the input exchange is identified by a correlation ID consisting of the Testrun ID and Test ID, and it is recorded into Elastic search. When accessing the Zebrunner web UI after a certain period – such as a week, day, or hour – users can obtain the relevant information by providing the Test ID or Test Run ID, querying the Elastic search via an HTTP request, and retrieving the necessary log data. This enables users to easily access the context of the data they need.
Logstash Pipeline Configuration
Create a Logstash file and configure the pipeline with input from RabbitMQ and output to Elastic search. The ELK stack is being used in a Docker context. Host is set to “elasticsearch”, it’s also a local host for Logstash, ES and LS are in the same container.
RabbitMQ configuration:
input
{
rabbitmq
{
host => "rabbitmq"
port => "5672"
exchange => "logs"
metadata_enabled => true
user => ”qps"
password=> ”secret"
durable => true
subscription_retry_interval_seconds => 5
}
}
ELK configuration:
output
{
elasticsearch
{
hosts => ["localhost"]
index => '%{[@metadata][rabbitmq_properties]
[correlation-id]}'
document_type => 'test'
}
}
To access Elastic Search from an external REST client, cross-origin request policies in Elastic Search YAML need to be adjusted. Set http.cors.enabled: true and specify the allowed hosts or use “*” for requests not tied to an origin host.
Elasticsearch CORS configuration (elasticsearch.yml):
http.cors.enabled : true http.cors.allow-origin : "*” http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE http.cors.allow-headers : "*"
Let’s talk briefly about streaming video now. Selenoid has useful plugins: enable VNC and video. With enable VNC, you can use a container to connect to a testing session in real-time and troubleshoot any issues through a VNC session. This is done by opening a port and connecting via a browser or VNC client.
Carina/Zebrunner – VNC Streaming (Web)
To enable VNC support in a test session, use the enableVNC=true command with a testing framework or Carina itself, which runs Aerokube’s Selenoid container. However, Selenoid does not support accessing this session via a secure web-socket protocol by default. To address this, use NGinX as an entry point from the outside, with Zebrunner or another reporting tool interacting with it through a secure protocol. NGinX proxies the traffic into the environment through an unsecured web-socket, using a remote framebuffer protocol as a client. This ensures secure interaction with pre-installed certificates for production environments.
Config uses GoGridRouter hub to aggregate multiple Selenoid instances. GGR distributes requests among nodes to run large test amounts. VNC parameters tell GGR how to proxy traffic from VNC point to Selenoid node. NGinX config allows secure socket access via WSS protocol to proxy traffic through VSS to GGR using unsecured protocol.
Selenoid configuration:
<?xml version="1.0"?> <qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru"> <browser name="chrome" defaultVersion="108.0"> <version number="108.0"> <region name="stage"> <host name=”host" port="4446” vnc="ws://host:4446/vnc"/> </region> </version> </browser> </qa:browsers>
NGinX configuration:
#GGR-WebSocket
location /vnc {
proxy_pass http://ggr_server;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header Access-Control-Allow-Origin
demo.zebrunnerqaprosoft.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
}
Carina/Zebrunner – VNC Streaming (Mobile)
VNC streaming for mobile devices enables interacting via a web interface. To automate mobile testing, create a Docker container with ADB, consisting of Java, Appium and STF.
STF connects to mobile devices via TCP, so you have to additionally install it which would let you proxy-pass the traffic from a web-socket type to a TCP-type.
To enable VNC streaming, use the plug enableVNC=true to start /websockifyproxy. However, the unsecured protocol requires NGinX and WSS for security. Use an RFB client to connect to the URL for real-time information on the tested mobile device.
Video Recording
All the logs are accessible in both real-time and in historical format. Concerning video, a connected VNC session is in real time. However, it sometimes happens that you later analyze the tests in perspective. You need to check 5–6 failed tests and watch the incident’s video recording. Every enterprise platform, such as BrowserStack, records the video so you can see what’s happening and the reasons why the tests failed. Enable Video is a standard capability of Selenoid. This means that Selenoid runs an additional container that proxies all VNC traffic and creates an mp4 video file.
Video Recording (Web)
To make a recorded video accessible via the web, set up NGiNX and Selenoid to use the same folder. Use the session ID to correlate recorded sessions with tests. Override the video output variable in Selenoid and transfer everything in the container “opt/Selenoid/video” to the current host machine directory “/Selenoid/video”.
Selenoid configuration
selenoid: network_mode: bridge image: aerokube/selenoid:latest-release volumes: - /var/run/docker.sock:/var/run/docker.sock - $PWD/selenoid/:/etc/selenoid/ - $PWD/selenoid/video/:/opt/selenoid/video/ environment: - OVERRIDE_VIDEO_OUTPUT_DIR=$PWD/selenoid/video/
NGiNX Configuration:
nginx:
image: nginx
volumes:
- ./selenoid/video:/usr/share/nginx/video
location /video {
root /usr/share/nginx;
}
NGiNX is set up in a Docker Compose and the Selenoid video volume is shared internally at user/share/nginx/video. The NGiNX content requires addressing resources in the /video folder statically.
Video Recording (Mobile)
In Appium 1.22.4, you can record a real application testing session on a mobile device or emulator by setting parameters for video resolution, bitrate, and providing FTP server credentials. Appium automatically uploads the video to the FTP server when recording stops.
A public Docker image can be used to set up the FTP server, and NGiNX and FTP point to the same folder. You can access the recorded video by using the regular scheme host/video/sessionID/mp4, where SessionID corresponds to the test run. It is possible to build a cost-effective TestOps infrastructure using only open-source products.
Suggestions for further reading
- DevFemOps: enhancing maintainability based on microservices using formal engineering methods
- http://zebrunner.github.io/carina/
- https://appium.io/docs/en/about-appium/intro/
- Defining TestOps: Collaborative Behaviors and Technology-Driven Workflows Seen as Enablers of Effective Software Testing in DevOps
About author
Maksim Sarychau is a Test Automation Solution Architect and QA Engineering Manager at Solvd, Inc. He took his first steps in IT when he was only 8. He has a master degree in radiophysics & artificial intelligence, worked as a system administrator, programmer and designer, and after taking a test automation course in 2003, got his first job in testing. He is a professional in Java, Ruby, C#, Node.js and Carina, to name a few, and has contributed to creating several new test automation frameworks.