Impact evaluation drives effective public sector change

Table of contents:

• Introduction
• The value in the impact
• Obstacles to impact evaluation
• 3-step framework for practical impact evaluation
• Conclusion

Introduction

The value of impact initiatives—purpose-driven efforts designed to create meaningful social or economic outcomes – is universally recognized by governments and non-profits. But how do organizations ensure that these public sector initiatives are genuinely effective? The answer lies in rigorous impact evaluation.

Impact evaluation is no longer just a nice-to-have; it's a critical process that helps jurisdictions forecast and anticipate the impact of programs and interventions and accurately measure outcomes.

Governments of all types have long been exploring ways to harness new types of insights that support data-driven and informed policy making. This kind of next-generation anonymized and aggregated data can be leveraged to track the impact of specific events or public realm changes across different industry segments in near real time.

Mastercard has formed public-private partnerships with governments and non-profits to measure the impact of their initiatives and investments through tools like Mastercard’s Test & Learn®. This blog highlights our work on two impact evaluation projects: New York City’s Open Streets holiday initiative and St. Louis' 911 Call Diversion program.

We also delve into the growing focus on impact evaluation, its rationale and the obstacles to overcome.

The value in the impact

Impact initiatives are purpose-driven, organized, time-bound efforts targeting a specific social or economic outcome. Government agencies, non-profit organizations or private entities in the public sector can carry them out. As public and non-profit spending comes under more scrutiny, the ability to clearly and quantitatively showcase impact is a growing priority.

Data-driven impact measurement is critical in assessing the effectiveness of government and NGO-led social and economic programs and provides accountability to donors and taxpayers. While it may be obvious that program owners would want to know if their initiatives are broadly effective, it is even more valuable to understand the main drivers of the program’s impact. Unravelling the 'how' and 'why' behind a success or failure is more helpful than a binary 'yes' or 'no' and can help stakeholders adjust programs to deliver more optimal outcomes.

Take, for example, NYC’s Open Streets holiday initiative in which select streets were closed off to vehicular traffic to boost sales at local businesses during certain days in December. Impact evaluation revealed the potential increase in spending resulting from the initiative and the drivers behind incremental spending for businesses from pedestrian traffic. Moreover, the evaluation confirmed that the extended pedestrianization hours led to higher sales.

The City of St. Louis was able to reduce dispatcher and officer hours spent on behavioral health cases with its 911 Call Diversion Program. Non-violent mental and behavioral health calls are routed to crisis counselors rather than police officers. The city effectively identified the program's value by leveraging a rigorous test vs. matched control methodology.

Here are four ways that impact evaluations can show the value of these types of initiatives:

• Evidentiary insights: Empirical evidence can show what works, what doesn't, and under what conditions.
• Sustainability assessment: The longer-term sustainability of initiatives can also be measured, providing more confidence in long-term outcomes.
• Improved accountability: Justify the financial investment in an initiative and help with future fundraising.
• Learning and decision-making: Individual and organizational-level learning is encouraged and allows decision-makers and donors to make informed decisions.

Obstacles to impact evaluation

The need for impact evaluation is straightforward. Still, several obstacles stand in the way of broader adoption:

• Funding & resource constraints: Impact evaluation studies can be seen as a ‘nice-to-have', leading to challenges in securing funding for data collection and data analysts.  
• Data availability & quality: Real-world data can often be noisy, low-quality and dated. Governments and NGOs often face difficulties in collecting accurate data that can be useful in assessing impact.  
• Time lag between intervention & impact: Interventions for public impact may take time to yield benefits, resulting in long delays before evaluations can be conducted and the benefit of the initiative assessed.
• Establishing baseline performance: The value of an intervention can only be determined if there is a clear understanding of the status quo, but establishing true ‘test’ and ‘control’ groups may not always be feasible.

Due to these challenges, governments and non-profits often struggle to measure the impact of their initiatives, preventing them from accurately forecasting impact and creating compelling business cases for essential funding.

While New York City had a sense that its holiday season pedestrianization effort was a success, conducting a comprehensive impact evaluation study wasn’t easy. The government tried to survey the local merchants, but the data gathered was sparse and biased. The uniquely commercial neighborhood setting made it even more difficult to analyze the initiative.

Similarly, when the City of St. Louis tried to test the effectiveness of its mental and behavioral health call diversion program, operators were unable to construct a reliable methodology to gauge effectiveness. Instead, officials had to rely primarily on anecdotal feedback, which lacked the necessary rigor to convince elected officials to renew funding.

3-step framework for practical impact evaluation

Impact assessments can be more straightforward with a structured approach, spanning objective setting, intelligence, optimization and design, and the buy-in of key stakeholders and decision-makers throughout. Building processes and partnerships can help any public sector or not-for-profit customer become a sophisticated impact evaluator.

Objective setting

The first step in any effective impact evaluation program is to identify the organization’s primary objectives. Input from program owners, sponsors and funders can enable an evaluation design that prioritizes the most relevant focus areas. Answering a few key questions can help define organizational objectives:

• What are the program’s goals?  
• What is the expected timeframe for these goals?
• Are there other concurrent initiatives planned?

The next step is identifying indicators of success. These can be short-term metrics, such as real-time data that would enable immediate quantification of impact, or long-term indicators that can provide a view into systemic change over an extended timeframe. These long-term indicators can include housing rates, job program success and economic development indicators.

Stakeholder involvement in setting objectives and performance metrics can differ depending on the organization's size and structure. In small organizations, stakeholders with signatory authority, such as executive directors, can champion the organization's commitment to evaluation. However, in larger organizations, data collection teams should partner with internal teams to collect accurate and unbiased data. Some organizations may even have dedicated learning & impact teams responsible for designing, executing and interpreting evaluation initiatives.

Intelligence, optimization and design

An assessment of the data available to the organization, internally and through external sources, must be conducted to evaluate any gaps in the dataset. Through the Intelligence phase, Mastercard’s insights database and location assets could complement the initiative owner’s data for a more comprehensive evaluation. Some of these unique datasets include the Inclusive Growth Score™, which provides local planners, governments and impact investors with a clear, simple view of social and economic indicators at the neighborhood level.

As a part of Optimization, initiative owners will then use the data identified in the intelligence phase to perform short and long-term impact assessments through data analytics. Depending on the research objectives, this can take the form of testing and optimization through platforms such as Mastercard’s Test & Learn® or through location-based insights that marry publicly available metrics segmented geographically with spend insights.

The outcomes of the analysis should then guide the future strategy of the government or not-for-profit organization. Through the Design phase, impact initiatives can be further tailored and prioritized for funding to ensure they serve their communities and meet the organizational impact objectives.

Through this multi-step framework, Mastercard continues to partner with organizations in its communities to help them successfully evaluate their impact. These partnerships have helped non-profits and government organizations effectively drive impactful results and design future impact initiatives.

NYC Open Streets

In 2023, Mastercard partnered with the NYC government to measure the impact of its 2022 holiday pedestrianization initiative, which drove $3 million in incremental spending. When the city first tried to measure the impact, it began by directly surveying local merchants. However, the resulting data was sparse and biased, and the uniquely commercial neighborhood setting made it even more challenging to analyze.

Mastercard set up the three-step intelligence, design and optimization framework and used our unique datasets to examine spending in pedestrian-only and adjacent city blocks compared to spending in similar city blocks that were not pedestrianized during that period. The optimization step then allowed NYC to measure the incremental economic impact of the initiative. The results included:

• Seeing a 6.6% increase in spending from domestic cardholders
• Identifying specific industries driving the spending growth—accommodations, apparel and eating places
• Determined the variable driving the most incremental spend (operating hours); stores that stayed open longer saw a disproportionately large lift in spend.

City officials were then able to shift to the last step, design and understand how it can best support small businesses by investing in economic initiatives, such as pedestrianization.

St. Louis 911 Call Diversion

In 2022, the City of St. Louis launched a 911 Call Diversion program to enable emergency dispatchers to route certain non-violent mental and behavioral health calls to crisis counselors at Behavioral Health Response, the city’s non-profit crisis provider. To ensure that elected officials continued funding the program, initiative owners needed to prove the efficacy of the program in improving both public health and public safety outcomes. That’s where Mastercard came in.

By leveraging Mastercard’s Test & Learn®, the city was able to measure the incremental impact generated through the 911 Diversion Program. The analysis found that calls determined to be mental health-focused and subsequently routed to crisis counselors saw an 80% resolution rate without police or EMS intervention, saving 900 police and ambulance dispatches and $400,000 just in 2021.

Before the implementation, most of the evidence was anecdotal and did not provide a complete view of the impact of the intervention. Wilford Pinkey, who leads the program at the St. Louis Mayor’s Office highlighted the value of having more robust data to rely on: “Now we have the data that tells us about the engagement, the outcomes and our ability to handle it. It lets us know what we learned about the follow-up and how we’re able to prevent repeat engagements. We know there are more calls that we could divert.”

Conclusion

The ability to quantify the benefit driven by an impact initiative can provide organizations with the evidence required to validate impact and justify funding. These kinds of data & insights capabilities empower governments to address challenges across a myriad of verticals including economic development, event impact, transit, tourism, small business, disaster preparedness & recovery, public safety, housing, health, utilities, and so on.

Even if there are gaps within a program today, it doesn’t mean that the path to assessing impact is closed forever. Instead, it presents an opportunity for future initiatives to be designed with the three key evaluation principles in mind - Intelligence, Optimization and Design - enabling a solid foundation for future impact measurement opportunities.

Learn more about how Mastercard works with governments or get in touch with a services specialist.