Minnesota expanded its internal capacity to evaluate programs and guide their improvement to ensure better public outcomes with the creation of the state’s Impact Evaluation Unit. Part of the Minnesota Department of Management and Budget (MMB), the unit was launched in 2019 as an innovative initiative by lawmakers who wanted to evaluate more state-funded programs to learn what works and strengthen the state’s response to the opioid crisis.
The evaluation team is currently conducting studies of taxpayer-funded substance use disorder interventions and social service programs to better understand their effectiveness and potential for improvement. Other states can learn from Minnesota’s investment in research capacity to develop evidence-based budget plans and strengthen resident services.
In 2019, the Minnesota Legislature ordered MMB to evaluate grant-funded social service programs to determine their effectiveness. The department then called on researchers to do the studies and establish the new unit.
“It’s great to have an in-house research resource,” said Neerja Singh, clinical director of behavioral health at the Minnesota Department of Human Services. Singh wants to see the impact of the new programs. “We have good inter-agency collaboration. It is becoming the norm to use MMB to plan the assessment each time we start a new program.
The legislature also directed the MMB to evaluate grant-funded services provided by the Opioid Epidemic Response Advisory Board, which oversees the state’s efforts to address the opioid crisis and the management of opioid use disorder (OUD). With approximately $5 million in annual revenue, the board provides grants to state and county agencies and private providers to provide TOU prevention, treatment, and recovery services.
The Impact Evaluation Unit focuses on evaluating council-funded programs that have not been classified as “proven effective”.
“The unit has been hard at work,” said State Senator Julie Rosen (R), patron of the legislation that created the unit. “They provide information that helps make our investments responsible. We want to know: will the program be worth the investment? »
A focus on effective practices to scale up evidence-based treatment
MMB has hired six full-time staff who specialize in scientific research methods and statistical data analysis, and who have evaluation expertise, to conduct the studies. Researchers covered a range of human services topics. For example, Weston Merrick, the unit’s senior director, said an upcoming project will look at the impact of changing practice in suspending driver’s licenses for non-payment of child support. Another will analyze efforts to connect people coming out of incarceration with Medicaid and other benefits they might be eligible for.
In August 2021, the unit completed its first evaluation of an opioid-focused initiative, Project ECHO (Extension for Community Health Outcomes), a telementoring and education program developed at the INHA Health Sciences Center. University of New Mexico. Project ECHO connects primary care providers with specialists elsewhere to offer training and support for drug treatment, including best practices for providing drugs such as buprenorphine for OUD. Researchers found that Project ECHO providers were more likely than similar providers to prescribe buprenorphine, so the project was considered a success.
“What was really exciting with MMB was using their vast data resources to really link what ECHOs are doing to tangible changes in practice,” said Dr. Gavin Bart, director of medicine at the substance abuse and ECHO team physician at Hennepin Healthcare, the system health care center that runs the program. “It has a downstream effect on patients, which is the ultimate goal. And we couldn’t have known that without collaborating with MMB. »
Findings from the evaluation of Project ECHO helped expand the reach of the program to more counties, including underserved areas. This was important because opioids are the leading cause of drug-related deaths, according to data from the Minnesota Department of Health. Additionally, historical data has shown that Native Americans in the state are up to seven times more likely to die from drug overdoses than whites, and African Americans are twice as likely.
“We know this is helping because we know more prescriptions are now being given to underserved patients seeking care,” said Dr. Brian Grahan, who leads the opioid-focused ECHO Project.
Last fall, Hennepin Healthcare partnered with the Native American Community Clinic to host the Midwest Tribal ECHO to help train health care teams to provide more culturally appropriate care to Native Americans struggling with the ‘OUD.
The team uses state administrative data and works with partner agencies to study the effects of the program
Conducting an impact study, such as Project ECHO’s, involves listening to program and agency needs, formulating actionable research questions with program stakeholders, and setting evaluation objectives guided by agency or program preferences. It also requires managing the expectations of program directors by indicating which programs are good candidates to assess.
“We have to say that we can’t promise that an assessment is possible, but we can help them see what we can study,” Merrick said. “Most program managers want to know what works and what doesn’t. And it can be difficult for them to refuse projects because an impact assessment is not feasible.
When an impact evaluation is not possible, Merrick said the unit can provide other technical assistance, such as helping to identify outcome measures from administrative data or referring program managers to other researchers who can do descriptive evaluations.
Once programs are selected for evaluation, the unit reports its study plans to stakeholders and other researchers—before collecting data or reporting results—by publishing its study proposals through a recording prior to the analysis. “Our review policy and pre-registration requirements help lawmakers see that we are nonpartisan,” Merrick explained. “And we don’t show reports that intentionally favor one agency or one vendor. Pre-registration is a means of enforcing the principles [in our evaluation policy].”
Unit staff members communicate regularly with state partners, sharing project updates and findings with commissioners, legislators and agency leaders. The team delivers the analyzed datasets to agencies and provides them with suggestions on how to use the data in the planning, monitoring and subsequent evaluation of programs and services. After the unit completes its studies, team members work with stakeholders to find ways to implement the findings to improve program performance or administration.
The unit can handle four to five projects at a time; the studies are complex and can take 12 to 14 months. Established inter-agency data sharing agreements help the unit reduce overhead costs that can significantly increase contract study costs. Under these agreements, unit personnel can work with agencies to access, prepare, and analyze data while adhering to security and confidentiality requirements.
As leaders in other states seek to invest pandemic-related funds from the American Rescue Plan Act and other sources, they should consider building their evaluation capacity to better understand what programs are working and to who.
Steve Lize works on The Pew Charitable Trusts Results First initiative.