Despite the proven value of collecting raw data from testing activities and analyzing it to create actionable, easily digested key performance indicators (KPIs), I find that most firms still struggle with utilizing test process data and organizing it into functional reports. Only a few have been able to implement truly effective, streamlined reporting systems—and many teams and their leaders don’t even recognize what they are missing.
In some cases, team members assume that because their various reports provide interesting information, they have gone far enough. Others are so bogged down with tedious processes such as manual data collection and report generation that they cannot envision anything beyond the next report.
This is terribly unfortunate, because the trail of data that testing activities leave behind can be incredibly beneficial for decision makers striving to improve test quality—and to reduce costs and accelerate release cycles.
From OK to Awesome
It is not useful, productive, or cost effective to have isolated teams and team members creating manual reports. To extract full value from test process data, organizations must implement a unified, standardized framework for data extraction, transformation, and loading (ETL), with report standardization and socialized visibility across the enterprise. Properly designed and implemented, a reporting framework can enable team members and leaders to monitor and improve testing activities across the board, including test case development, test planning and execution, defect removal, and more.
Otherwise, it is indisputable that these teams and the organizations for which they work are missing opportunities to effect meaningful, positive change driven by fact-based intelligence. The question then becomes, “How does an organization accomplish this goal?” There are several steps involved in transforming test process data into actionable, easily digestible, insightful information.
Over the next few months, we will discuss this journey and provide insight not only into how an organization can implement a functional reporting framework but also how to properly use the available metrics and KPIs to effect quantifiable, continuous improvement. This month, we will discuss the first step: establishing where the company currently stands, in terms of reporting sophistication so leadership can set realistic goals for positive change.
Find Your Place
In our work helping organizations make this journey, we have identified five stages of reporting sophistication, which range from chaotic to predictive. In my experience, a considerable number of organizations are stuck at the chaotic stage, where the standard for data gathering is manual, isolated, individual reporting. Others have moved to later stages—some have adopted standardization to achieve automated report generation, and a handful are successfully interpreting their reports and using them to monitor projects and manage risk.
A firm’s current position is a predictor of how much effort will be necessary to make the leap to reporting excellence. Although it’s undeniable that firms at the most fundamental, disorganized level will have the farthest to go, the good news is that they can gain dramatic benefit along the way—long before they complete the entire journey.
Level 1: Chaotic
At this most basic level, team members create reports (usually manually; sometimes with basic tools) to suit their individual needs. Data gathering, metrics, and report standardization across projects are almost always non-existent.
Level 2: Sporadic
Here, team members create reports (manually with basic tools or through limited automation) to fulfill the needs of local teams. Some reporting standardization may be in place, but it isn’t widespread or fully implemented.
Level 3: Reactive
The use of standardized tools, data values, metrics, and workflows enables automated generation of uniform, domain-specific reports. Functional leaders may be able to share this information with project stakeholders, but it’s unlikely that the information is being used for informed decision making and improvement.
Level 4: Proactive
Reports are being developed from well-defined, readily available metrics and KPIs, and the organization is reaping tangible value from them. Project leadership is relying upon the reports to monitor project statuses, prioritize efforts, and identify and manage risk. However, enterprise-wide visibility and ongoing improvement through historical analysis and feedback loops are still missing.
Level 5: Predictive
Standardization of ETL processes, as well as standardization of reports and reporting, has been achieved across all initiatives and projects. Teams follow a disciplined approach, leveraging and continuously refining KPIs. Historical data drives forecasting, with trending and analytics providing deep insights and enabling data-driven action. The entire enterprise benefits from socialized KPIs, gaining fact-based confidence and improving project efficiency and outcomes.
In our experience, a large percentage of firms fall into one of the first three categories. Those who make it to the reactive or even the proactive stage frequently and inaccurately believe they have gained sufficient value from data and stop their efforts there. The ones who recognize there is additional benefit to be gained are often uncertain how to achieve it.
Until an organization can achieve enterprise-wide visibility and ongoing improvement through historical analysis and feedback loops, it is not reaping full value from data ETL and the reporting that the data make possible. No matter where organizations fall in their journey to implementing functional reporting systems, nearly all of them are not extracting full value from the data that is readily available for collection and analysis.
Over the next few weeks, take a hard look at how reporting is being handled in your organization—not only by you or your team but by others, as well. You may discover that other teams have achieved a higher level of reporting sophistication, and they may be able to provide tips or insight that will help you emulate their success.
Once you have a good idea of your team’s (and your organization’s) level of reporting sophistication, stay tuned. Next month, we’ll outline the six-stage journey to developing a functional reporting framework. Later on, we’ll dive into metrics and KPIs to help teams gain clarity and eliminate confusion about the data with which they are working.
Orasi offers an automated reporting solution, Orasi Automated Reporting Solution (OARS), that automates the extraction, transformation, and loading of data into pre-defined templates. It provides programmatic recreation of metrics and KPIs in the form of consumer-friendly dashboards, scorecards, and reports, helping firms make the most of their test data and transition from chaotic to predictive. To learn more, we invite you to access the data sheet or view the webinar recording.