Over the past several years (2016-2022) UNITAR’s outreach to learners has increased significantly, including participants in learning events containing an objective assessment of learning. However, the proportion of certificates of completion (CoC) issued to these learners has declined. In 2023, the UN Board of Auditors observed this trend and recommended UNITAR to undertake an evaluation to uncover any factors driving completion rates. 

This evaluation assessed the effectiveness of UNITAR's learning-related programming, focusing on factors driving learner completion and certification rates. It covered learning related UNITAR programming implemented from 2016 to 2022. It followed a mixed-methods approach drawing from participants and UNITAR personnel surveys and focus group discussions, key informant interviews with UNITAR personnel and other learning service providers, a document review, and data from UNITAR's Event Management System (EMS).

Challenges encountered by the evaluation included i) limitations in the recording procedures of EMS data; ii) potential recall bias for participants taking part in events more than four years ago; iii) limited information from UNITAR staff at managerial levels; iv) time constraints to delve into further data, such as Moodle analytics. Mitigation strategies were implemented where possible.

Motivation v. demotivation: What motivates enrolment and completion? The evaluation found that participants are primarily motivated to enrol in UNITAR programmes by their desire to acquire new knowledge and skills and interest in the porgramme topic and its usefulness to their careers. These same factors added to their personal commitment for growth motivated participants to complete the training. On the other hand, time constraints, workload from other responsibilities and unforeseen life events were the main obstacles to completing the event. Participants from countries in special situations were driven more to enrol and continue in the programme by present demands compared to their counterparts. 

The evaluation identified several existing good practices to motivate and support learners completion, although these are not applied systematically throughout UNITAR. 

Factors affecting performance: Which factors are correlated to completion rates? The evaluation uncovered factors with a positive correlation with completion rates which include in-person and blended learning events in contrast to online learning, however completion rates for online learning increased when the event duration is longer. For in-person events, shorter events were related to higher certification rates, excepting fellowship programmes. Events with selection processes also exhibited higher CoC rates. Participants characteristics were also related to completion rates, with male participants showing higher completion rates than women in online events but lower for in-person training. Participants from countries not classified as in special situations have higher completion rates for in-person events, and for overall events, but application rate was lower when engaged in jobs in NGOs or international organizations. 

Differently, factors with a negative correlation to completion include the COVID-19 outbreak, which also impacted the overall satisfaction and job usefulness metrics reversing the correlation with completion rates; donor-funded events, and events delivered as webinars. Participants with disabilities had lower completion rates than participants without disabilities. 

Application of knowledge, intended and unintended outcomes: Do application and changed behaviour change depending on certification status? Evaluation findings indicate that both participants who complete requirements and those who do, have applied knowledge and skills from the programmes and contributed to SDG implementation. Application of inter-personal skills is however higher for those completing requirements. Likewise, both groups reported intended and unintended changes. 

EMS recording procedures: What are current CoC recording procedures? The evaluation revealed discrepancies in EMS data recording impacted conclusions. Differing practices, interpretations and familiarity with policies and systems were found across units.

Industry standards: What are current certification practices in the learning industry? It was found that other learning institutions define certificate types, awarding criteria, and completion targets differently. Factors such as content alignment, learner preferences, cost, and format impact rates. Organizations use multiple success metrics and some have robust monitoring systems enabling analytics.

Five recommendations were issued by the evaluation:

1. UNITAR Programme Units delivering learning events should ensure full compliance with the UNITAR Certification Policy and strengthen familiarity of programme unit personnel with the Certification Policy and Objective Assessment of Learning (OAL) guidelines.
2. UNITAR Programme Units should integrate more rigorous and consistent quality control measures and standardized procedures for data entry into the EMS to ensure accurate data recording and analysis, including avoiding instances where the CoC rate is inaccurate or zero.
3. The Communications and Information Technology Support Unit (CITSU) should prioritize the development of the application programming interface for integration of the certification data from Moodle into the EMS, as planned in 2024.
4. UNITAR Programme Units delivering learning events should continuously formulate learner engagement strategies, particularly for those activities funded through donor-supported projects, and provide individual learner support, learning from the good practices in this report. Such practices need to be adjusted depending on the size of the cohort. 
5. UNITAR Programme Units delivering learning events should use evidence from learner analytics and evaluations to design or adjust future events in a way that maximizes learner engagement and increases completion rates. 

Four lessons also resulted from the exercise: 

1. Acknowledge varied perceptions of objective assessments of learning. Appreciating and accommodating differing perspectives can foster greater learner engagement.
2. It is imperative to recognize that the design, delivery, and content offered by UNITAR training holds intrinsic value comparable to the UNITAR certificate itself.
3. Prioritizing strong UNITAR branding for joint initiatives is essential to increase visibility and recognition.
4. Recognizing the importance of enhancing information management through high-quality, standardized, user-friendly systems is crucial to ensure accurate data recording.