Mining Gold from User Clicks: How Product Analytics Transformed Our Quality Strategy

Product usage analytics are an untapped gold mine for software quality assurance teams. This article shows how linking user behavior and connecting to quality strategy through knowledge graphs, applications can improve both technical and business quality.

Author: Vignesh Govindarajan Ravichandran

The Wake-Up Call

At almost three in the morning, my pager blared with the dreaded message: “Application Health Check-Failure.” This was the third major system outage in two months. I had to log into my PC in order to gain some perspective on the situation and grapple with the uncomfortable truth: our classical quality assurance approach had, yet again, failed to catch the faults that were causing extensive downtime and irretrievably lost business opportunities.

In the subsequent root-cause analysis, the respective module Quality lead stated “But we had 92% test coverage; All critical paths were tested.”

This moment changed everything about my thinking concerning software quality. With nearly 100% test coverage, we still had failed to capture what really mattered: how our users navigated and interacted with the system in real-world use.

Beyond Test Coverage: The Quality Analytics Revelation

In financial services, a single calculation mistake can lead to compliance infractions or huge financial losses. Against this backdrop, traditional or conventional quality metrics such as test coverage or defect rates, in some cases, can provide only a partial insight. Rather the true quality of the software is a measure of its capability to satisfy user needs and expectations for any application with respect to how those requests or expectations may reveal themselves in terms of usability and client satisfaction/experience.

Thus, our journey started with a very simple question: what if we could turn the product usage data into actionable quality insights? In collaboration with our product analytics team, we have developed a more detailed framework to capture, analyze, and visualize the use of our risk management platform in real scenarios. The findings were deeply distressing:

• 72% of users completely skipped our painstakingly well-defined operating journey for risk profiling
• Performance reporting – the most tested feature – was used by only 11% of our users
• 83% of customer support tickets stemmed from four minor user journeys that were given very low testing focus

The most compelling was the identification of a strong linkage between defined usage patterns and client churn. Customers who abandoned specific workflows or had especially elongated response times in key transaction paths were 4.6 times more likely to either reduce their usage or walk away from the app onto alternative applications/data sources rather than raise a ticket within a period not exceeding six months.

Building the Knowledge Graph

We built a knowledge graph that established connections between usage behavior, system performance, error states, and business outcomes. This knowledge graph serves as the basis for our predictive churn model as well as quality transformation.

The knowledge graph approach contributed to several advantages over conventional analytics:

1. Relationship Visibility: Rather than treating isolated metrics as a basis for quality decisions, we could visualize how particular actions performed by users infused with certain metrics can link to business outcomes under clearer quality priorities.

2. Pattern Recognition: If not for the process of mapping action sequences, the tendency of the graph to reveal missing workflows would have gone unnoticed by individual events tracking.

3. Predictive Capability: Authoring predictive behavior patterns that helped in determining at-risk clients prior to them actually entering the churn zone was possible with the help of the graph.

The broader platform that underlies our knowledge graph entails the following five steps:

1. Event Sequence: This list comprises user workflows across the system to include both functional (existing flows) and additive (new workflows due to latest changes scheduled in the release).

2. Performance of each Node: Next, it is the identification of performance metrics for specific steps in user journeys. These might then bring to light some steps whose longer response times might affect critical paths.

3. User Action and Error Correlation: Positioning relationships between user actions and error states- when users don’t report problems- is Step 3.

4. User Abandonment Pattern: After the above processes, inspect product analytics to observe screens/fields where users persistently abandon processes: indicators for usability or value perception issues.

5. Measure Business Impact: Finally, build links from the technical patterns back to business outcomes: support costs, renewal rates, and feature adoption.

There were a few other hurdles in the way of implementation; privacy aspects, handling data volume, and integrating with already existing monitor systems became a challenge. Nevertheless, so much value had been envisaged for this project that executive sponsorship was acquired for a dedicated cross-functional team to bring it to life.

Transforming Testing Strategy

With the plethora of experience gained from the knowledge graph, we completely revised our approach to quality assurance. The breaking changes in the QA strategy can be summarized succinctly in five components:

1. From Coverage to Impact

Instead of blindly pursuing arbitrary code coverage requirements, we employed a model where tests would be allocated according to how often different aspects of our application were used and how important these aspects were to the business. Business-critical path coverage became the new standard for success: giving heavy testing to those workflows that were the most central to client retention and satisfaction. For the clients, this translated to about 60% of testing done against paths related to risk calculations with more permutations of account types: the paths that the users spent the most time on and where the business impact of faulty testing was the most serious.

2. From Hypothetical to Actual Scenarios

We no longer employed generic test cases; instead, we created tests directly based on real user behavior. By replaying actual usage sequences of our most valuable clients, we ensured that our testing would be aligned more closely with real interaction patterns rather than idealized workflows.

In doing so, we unearthed that a lot of our clients in asset management utilized our portfolio re-balancing capabilities in a manner that we never anticipated – building multiple scenario analyses before actually executing trades. Traditional test cases entirely missed spotting this behavior. From Equal Treatment to Risk-Based Prioritization. The knowledge graph allowed us to derive risk scores for various components and paths based on their correlation with client churn and support incidents. Now, we would test not all code equally, but rather test a combination of code paths to different extents based on risk profiles.

3. From Equal Treatment to Risk-Based Prioritization

The knowledge graph allowed us to provide risk scores for various components and pathways based on their interactions with client churn and support incidents. Testing intensity was now dramatically differentiated based on risk profiles instead of equal weight being given to all code paths.

4. From Separate to Integrated Monitoring

This phase helped us to break down the wall between production monitoring and pre-release testing. Test cases were influenced by production performance and error data as they were generated and prioritized in real-time, representing a feedback loop.

5. From Technical to User-Centered Metrics

Possibly, the most important of all was that user experience, instead of technical correctness, was into the ground of how we defined our quality metrics. User completion rates, time-on-task, and a decrease in the number of support tickets were success indicators instead of the presence of defects.

The Results: Beyond Reduced Churn

Beyond Reduced Churn After having our quality strategy based on our knowledge graph running for 18 months, the results came in above what we had hoped to achieve:

• Client churn reduced by 32%.
• Software-related support tickets were reduced by 47%.
• Release velocity, higher than expected, went up by 28% with a corresponding harder test effort.
• Net Promoter Score increased from 24 to 41.

One very interesting result was the discovery of a hike in the unknown major user path. Our analytics showed that asset management clients needed to be able to produce compliance reports immediately after portfolio rebalance very often something that connected two workflows that our product team had never connected. After optimizing for this path and delivery, we saw a 74% increase in the usage of our compliance reporting feature and vastly improved client satisfaction scores.

Conclusion: The Untapped Gold Mine

Product usage analytics represents an untapped gold mine for quality teams. By linking user behavior and connecting to quality strategy through knowledge graphs, applications can dramatically improve both technical quality as well as business quality. Re-imagining the classical quality approach will need investment in analytical capabilities and breaking down silos between quality and product and consumer success teams. Engineering leaders need to transform silos into collaborative work norms to reap the rewards of less churn, better efficiency, and stronger client relationships. Investing In Quality Product Usage Analytics Teaches Us To Listen First Before We Build Better Any Thing. Listening to users’ behavior allows us to build better software and not just tests.

About the Author

Vignesh Govindarajan Ravichandran leads Quality teams for Risk, Analytics, and Investment Science at a leading asset management firm, ensuring resilient financial systems. An ASQ Financial Services Community Leader, Test Automation Summit 2025 speaker, and Gartner Peer Ambassador, he explores testing, DevOps, and finance.

References

Agarwal, R., & Dhar, V. (2023). Beyond the click: Data-driven quality engineering. Harvard Business Review, 101(3), 112-120.

Ebert, C., & Cain, J. (2022). User analytics in financial software quality. IEEE Software, 39(5), 36-42.

Financial Services Quality Consortium. (2024). Annual benchmark report: Quality in asset management software. FSQC Publications.

Huang, J., & Zhang, T. (2023). Knowledge graphs for predictive quality. Communications of the ACM, 66(8), 78-85.

International Organization for Standardization. (2023). ISO/IEC 25022:2023: Systems and software Quality Requirements and Evaluation (SQuaRE) – Measurement of quality in use. ISO.

Kim, G., Humble, J., Debois, P., & Willis, J. (2021). The DevOps handbook: How to create world-class agility, reliability, & security in technology organizations (2nd ed.). IT Revolution Press.

Malhotra, R., & Shah, P. (2024). Client retention through quality engineering. Journal of Financial Technology, 12(2), 145-162.

Singh, A., & Johnson, M. (2023). Graph-based analytics for financial software testing. Proceedings of the International Conference on Software Testing, 234-248.

Viswanathan, K. (2024). The new quality paradigm: From defect detection to business value. Financial Software Press.