The “Strengthening Road Safety – A Partnership to Build Capacity, Drive Innovation and Deliver Meaningful Impact” project, with the support of AB InBev and in collaboration with academic institutions and other private sector partners, aim to contribute to advance road safety targets by enhancing capacity of local and national authorities implementing road safety interventions, creating awareness on best practices contributing to road safety targets, and incorporating digital transformation to achieve road safety changes. The project is currently in its second phase. The evaluation covers the 2018-2021 phase of the project.

The evaluation’s purpose is to identify what went well and any challenges that the initiative has encountered; to issue recommendations, and to identify lessons to be learned on design, implementation and management to guide future phases of the project as well as the remaining period of the current phase. This is done by applying the six evaluation criteria of the Organization for Economic Co-operation and Development’s Development Assistance Committee (OECD DAC): relevance, coherence, efficiency, effectiveness, impact and sustainability.

Using a participatory and mixed methods approach, the evaluation included the review of existing project documents and a web-based search; interviews with project key stakeholders (CIFAL management, donor, participants in learning events and local and national government officials in partner countries); a survey deployed to beneficiaries and an outcome harvesting and validation of the project Theory of Change workshop.

The evaluation faced some limitations that restrict the project assessment and review. First, the COVID-19 pandemic prevented any field visit so that the evaluator could observe first-hand dynamics of the events. Second, the evaluation does not include a cost effectiveness analysis since the evaluator did not have access to the relevant financial data and had difficulties to measure impact at this early stage of the project, also considering that the COVID-19 pandemic has changed the dynamics of travel and mobility. Finally, resource constraints and information delays did not allow to rigorously compare the non-learning events results with learning events.

Using a four-level scale for assessment (limited, modest, substantial and excellent), the project was found to be modest and substantial across the evaluation criteria.

The project was substantially relevant to the beneficiaries needs and strongly aligned to UNITAR’s contribution to the achievement of the 2030 Agenda and SDG 3.6. However, the project can still improve in covering the needs of countries in special situation, women, and other groups made vulnerable.

Although strongly aligned to the 2030 Agenda, coherence of the project was modest, considering external actors operating in the road safety field. This, nevertheless, represents an opportunity to improve the project’s strategic direction, establish global linkages, and enhance inter-CIFAL collaboration.

On effectiveness, the evaluation found that while outputs were substantial, outcomes were modest. A rigorous results framework was missing to contrast the project achievements. Despite the COVID-19 pandemic, outputs delivered during the project phase were successful, including the learning related events and specialized offerings.

Furthermore, the COVID-19 setting required to change many events from face-to-face to an online delivery format. The evaluation observed that virtual learning, though it can reach more people at a lower cost, may face some constraints delivering training in developing countries. Financial arrangements were considered sound and prudent. The evaluation, then, found the efficiency of the project to be substantial.

The project’s likelihood of impact was assessed as modest because of the lack of evidence. Although some efforts to get the necessary feedback are being made, appropriate mechanisms for monitoring and evaluation with a suitable budget need to be formalized and implemented, inclusive of the Safer Roads Toolkit and the case studies used in the toolkit.

The AB InBev partnership demonstrated the potential for scaling up UNITAR’s activities in road safety with a suitable international partner. However, it showed the need for UNITAR to diversify and have a range of such partners to ensure sustainable outcomes. Also, beneficiaries’ impact from the project seems to be sustainable. The sustainability for the project was assessed as modest.

Based on the above findings, the evaluation issued six recommendations:

On overall strategy and reaching the furthest first:

R1: Develop a strategy that responds to the needs of low-income and other disadvantaged countries and that also addresses an appropriate mix of face-to-face and e-learning technology, customized to the needs of the beneficiaries.

R2: When developing the strategy, differentiate features of the project’s toolkit from other toolkits and emphasize the value added; link to other toolkits that provide additional information.

On coherence and partnerships:

R3: Actions to strengthen collaboration are: map who does what in road safety – identify synergies, overlaps and duplications; scale-up collaboration and diversify partnerships: e.g., collaboration with multinational companies should be pursued strategically; and integrate and scale-up data gathering with other partners.

On results framework, monitoring and reporting:

R4: Base the results framework on the Theory of Change with clearly formulated objectives, adequate baseline data and realistic targets, and report performance accordingly.

R5: Ensure follow-up surveys after project activities have ended with a view to measure key matrices (reaction, learning and application).

On gender equality and women’s empowerment:

R6: Intensify efforts to reach out to women and other vulnerable groups to ensure a gender-targeted approach for the remaining project duration, and incorporate a gender-responsive strategy for future phases of the project.

The evaluation identified four lessons learned from the implemented phase(s) of the project:

L1: Following the COVID-19 pandemic, strategize other potential shocks and outline mitigatory measures.

L2: The training needs in the most vulnerable low-income countries may require a dedicated funding mechanism from the private sector and developed countries.

L3: Because of the two-year project lifespan, long-term relationships with partners are essential to ensure sustainability after the project closes.

L4: In addition to a rigorous results framework, a coherent strategy and the selection of events that can influence decision-makers are key priorities to optimize resources.