A cluster evaluation of six UNITAR training of trainer (ToT) related projects implemented between 2011 and 2016 was undertaken to assess the relevance, effectiveness, efficiency, impact and sustainability of programming to strengthen capacities of learning centres in developing countries. The primary purpose of the evaluation was to identify factors contributing to or hindering successful implementation and achievement of results, and to issue recommendations and lessons on how to strengthen capacities of learning centres, including the identification of approaches that work well and the reasons why.

Using a mixed methods approach, the evaluation included an online survey to beneficiaries and key informant interviews and focus groups with UNITAR staff, partners and beneficiaries, along with a theory of change (ToC) reconstruction.

The evaluation had several limitations, such as the absence of or underdeveloped ToC and results frameworks to assess project effectiveness and impact. Assessing efficiency proved difficult as verifiable costs for the ToT-specific elements of the projects were not available. While it was not the evaluation’s intent to undertake a comprehensive evaluation of each project, the cluster nature of the evaluation and the differences in size, scope and purpose of the projects made generalization nevertheless challenging. Finally, at the time of data collection and field work, two of the projects had not yet completed all planned activities, and there was on-going discussion with both projects for either no-cost extensions or subsequent phases.

Relevance, alignment and complementarity                                                              

The evaluation found much alignment with the overall UNITAR strategy and programme of work and relevance to learner and institutional beneficiary needs. The evaluation also found complementarity with the UN, regional and/or other international as well as national strategies and policies for the training sector.

Results frameworks and risk mitigation                                                             

Results frameworks and risk mitigation plans were found to be missing or inadequately developed. Consequently, project documents did not provide stakeholders with a common understanding how ToT initiatives would develop the capacities of learning centres and what potential risks could mitigate the achievement of planned outcomes.

Quality, client orientation, ownership and approaches to learning 

The evaluation found ToT aspects related to quality, learning, client orientation, ownership, flexibility and knowledge-sharing were well established and widely acknowledged, yet a common UNITAR approach to ToT was lacking. While the projects in this cluster were distinct, all six comprised a clear ToT dimension with sessions, modules or entire workshops devoted to training needs analysis, instructional design. A review of documents revealed the lack of an overarching competency framework for trainers and that learning objectives from the ToT specific events uncovered many different learning objectives, some having a degree of commonality, yet were formulated differently.

Learner self-assessment

Except for one or two ToT specific events, achievement of learning objectives was based on subjective self-assessment by learners. While the results of self-assessments of learning clearly indicated an increase in knowledge and skills and while self-assessment provides certain advantages such as learner engagement, certifying training competency cannot be done using learner self-assessment alone. 

Entry and exit strategies

One of the overarching objectives of ToT initiatives is the development of endogenous capacities of learning centres to respond to needs and to decrease reliance on training expertise from the outside. Enhanced capacities of learning centres should, ultimately, translate into decreased reliance on the capacities of external learning centres as service providers. In fact, in one of the projects of the cluster, one year following the delivery of the project, UNITAR was called back to deliver the training. None of the six projects included a clear entry or exit strategy which communicates a common understanding on the role of UNITAR and the partner, at project start up to project conclusion and follow-up. The lack of an entry and exit strategy could compromises the sustainability of results.

A promising “niche market”

UNITAR’s institutional mandate as a global service provider of training using both face-to-face and online delivery methods provides a promising backdrop to develop a ToT niche market. At the time of the evaluation, however, this market was very nascent from an institutional perspective, despite various ToT-related projects and stand-alone courses. UNITAR programme units have pursued ToT through their specific approaches in the absence of much knowledge-sharing or a coherent institutional approach.

Based on the above findings, the evaluation issued seven recommendations:

Recommendation 1: Theory of change and results frameworks

UNITAR should ensure that future ToT projects contain a clear ToC and results frameworks with measurable and objectively verifiable indicators at the levels of outcomes and impact, based on needs and baseline assessments. It is also recommended to ensure that results frameworks include assumptions on moving from one level of results to another, and risk mitigation plans are developed and communicated to project stakeholders.

Recommendation 2: Standard competency framework with harmonized learning objectives

UNITAR should develop a trainer competency framework with standardized learning objectives, methods and assessment criteria for a ToT certification which can be used across all UNITAR programmes, with the understanding that some degree of customization to cater to the particularities of the targeted beneficiaries would be required.

Recommendation 3: Formalize certification of competencies or another recognition mechanism

UNITAR should develop a common objective assessment tool to include knowledge and performance-based testing to certify ToT competencies (knowledge, skills, behaviors/attitudes) and ensure that the tool is applied consistently across ToT programming.

Recommendation 4: Entry and exit strategies

UNITAR should ensure that all ToT projects contain a well-defined entry and exit strategy in consultation with the beneficiaries and donors, clarifying understanding between UNITAR, the learning centre and the ToT beneficiaries at the start and conclusion of the project, and a strategy for disengagement of UNITAR over time. This is important to manage expectations, enable sustainability and provide a benchmark for UNITAR engagement and measuring and evaluating the change that UNITAR services have provided to its clients.

Recommendation 5: Programme lead and promoting a niche market

UNITAR should assign a ToT programme lead at UNITAR to coordinate the development of a common approach to designing and delivering ToT related programming and use this coordinated approach to leverage UNITAR’s mandate and experience in developing a niche market to enhance visibility of UNITAR expertise.

The evaluation identified the importance of assessing needs and ensuring medium to long-term institutional engagement, as well as partner ownership, buy-in and clarity of roles as important lessons.