[author: Kyle Welch, George Washington University]
In 1969, computer systems company Information Concepts Incorporated transformed the world of baseball with the creation of “The Baseball Encyclopedia”. The book (affectionately nicknamed “Big Mac” in homage to both its weight and its publisher, Macmillan) gave the sport its first fully comprehensive and rigorously researched compendium of baseball statistics – and inspired a generation of fans interested in baseball history and statistical research. In 1971, sixteen of these “statistorians” formed the Society for American Baseball Research (aka SABR or “Saber”) and began using the Big Mac to develop innovative new measures to compare players and predict outcomes. In 1980, this practice had received a new name: sabermetry.
Today, sabermetrics is an integral part of Major League Baseball. Virtually all MLB teams employ sabermetricians, who replace “experience” and “gut feeling” with analysis of empirical data. The approach has allowed teams like the Tampa Bay Rays and Oakland Athletics to set winning records on modest budgets (most famously illustrated in the 2003 book and 2011 film silver ball). This disruptive quality, along with its tendency for counter-intuitive maxims, has helped this “big data” approach to decision-making capture people’s imaginations.
A similar wave of change has begun in line with the data. For a long time, mainly due to data limitations, there has been little examination of the causes and effects of the various efforts in compliance systems. Worse still, there was no evidence of what the compliance data told us about these systems. For example, are high volumes of hotline reports a good thing, illustrating an employee’s willingness to speak up, or a signal that the organization had deeper issues? Without additional information, this single stat cannot tell the story. As a result, compliance officers have largely had to rely on their own experience and intuition when interpreting the data. While these can provide valuable information, they also open the door to incorrect assumptions and personal biases.
Decision and data uncertainty are beginning to change in compliance. We are entering the golden age of understanding the insights provided by compliance data to uncover the signals and causal mechanisms of compliance.
Those at the forefront of this movement are giving birth to rigorous new data protocols in compliance, enabling compliance officers and organizations to more easily contextualize risk signals and better predict outcomes.
In other words, hotline reporting data is an extremely valuable set of information for compliance officers and executives. Reporting information is the pulse of organizational culture and should be weighed, analyzed and processed accordingly.
The counterintuitive insights of data for compliance
A few counter-intuitive ideas have emerged from initial efforts in this area. One of the most commonly noted is that companies with more actively used compliance reporting programs—those that receive more reports per employee—score “better” in almost every measure (i.e. more profitable, better governance structures, less negative media coverage, etc.) .
Additionally, organizations with the highest volume of reports per employee were the least likely to face lawsuits and fines; and those who did paid less on average than their peers in fines and settlements.
These measures contradict the widely held assumption that more reports indicate more problems.
This may lead to the question, “What does the data tell you about the quality of reports as the volume of reports increases?” Compliance officers are certainly aware of the misuse of feedback systems and can rightly be concerned about wasted time and resources on bad faith reports. Also, as with most business activities, there are almost always diminishing returns at larger scales of investment. The question is, has this happened with feedback systems?
In a crude analysis of report quality, we measured the prevalence of two factors as report volume increased: named (vs. anonymous) reports and report completion. Previous analysis of report data has demonstrated that reports that include the identity of the reportr and those that contain complete information (e.g. fields including direction involved, duration of event, how been discovered, etc.) were more likely to be useful or informative, or less difficult to review. Admittedly, it is unclear how important each individual report is in these two categories, but there is a clear difference in measuring quality by providing their own identity for follow-up and including standard information about the reported issue. .
Ex ante, we would probably expect the quality of reporting information to decrease as volume increases. As more and more individuals make reports, it seems natural that there are more issues and limitations with the users who provide this data. The graph below shows that this is not the case.
In the chart, we separate report volume by quintiles, with 1 being the lowest 20% report volume in a given year of the business and 5 being the highest report volume (the lowest 20% high), taking into account industry and other company factors.
Instead of finding what we would assume, we observe that the informational quality (i.e. completeness) of reports increases with the volume of reports. This trend, combined with other evidence, is consistent with the claim that companies with higher reporting volume are likely to 1) have more training and information on the effective use of reporting systems , resulting in 2) higher levels of information in these reports and 3) more problems being discovered before they escalate.
This is just one example of the counter-intuitive insights emerging from this field that are changing assumptions about compliance and employee feedback systems. The forefront of the compliance industry is exploring what I call compliance sabermetrics – the use of big data to provide additional information to management. These efforts lead management to change its view of compliance.
According to the old management mantra, when an audit committee has observed above-benchmark employee feedback through its reporting system, it may have mistakenly asked itself, “Why did we more problems than our peer societies?” The new wave of insights from data will prompt a different question to ask in the future: “Do we have the resources to ensure that we are effectively investigating the increase in employee insights?”
Data from internal reports should be treated as the wealth of information that they constitute. Creating a culture that allows for honest reporting and proper follow-up to correct issues should be a priority for management.
The growing collection and analysis of compliance data will further challenge long-held assumptions about which metrics deserve attention and what they say about a company’s organizational culture and health. Successful companies will invest in these efforts, placing less emphasis on intuition in favor of analyzing empirical data.
Download the full Top 10 Risk and Compliance Trends 2022
See the original article on Risk & Compliance Matters