The IOMC Toolbox for decision making in chemicals management – Phase III: From design to action project aims to support the implementation of SAICM by enhancing the identification and implementation of guidance materials for chemicals management by developing countries and countries in transition using resources developed by IOMC partner organisations. It also continued to improve the functionalities and broadened the scope and application of the Toolbox as developed in the two previous phases. The project was implemented between December 2017 and October 2022. 

The purpose of the final evaluation is to assess the achievement of the project’s planned Phase III results with respect to the project’s relevance, coherence, effectiveness, efficiency, likelihood of impact and likelihood of sustainability. It also identifies lessons from project implementation to inform decision-making for Phase IV.  

The evaluation followed a participatory and mixed methods approach, including carrying out an after-action-review with participating organisations’ (POs) focal persons, in depth interviews, construction of case studies, an on-line survey, desk review of project documents and using an artificial intelligence chatbot (ChatGPT) as in input into the answers to some of the more general evaluation questions.  

The evaluation encountered four limitations, namely; i) the difficulty of providing comprehensive case studies, ii) gaps in institutional memory and other issues with the turnover and transitioning of project personnel, which made it challenging to gather information, iii) misalignment in timing between Phase III evaluation and Phase IV development, and iv) unavailability of final narrative and financial report at the time of the evaluation.  

The evaluation found the project to be broadly relevant to global commitments and coherent with policies, programmes and projects at different scales. The project also achieved and exceeded most of its output targets (6 out of 7 indicators) in a timely and cost-effective manner despite a few setbacks including the COVID-19 pandemic and implementation issues. Meanwhile, expected impact of the project and likelihood of its sustainability are moderate. Some noteworthy findings include:  

• The project is the only platform that brings most of the IOMC POs to work together, and as such provides a unique opportunity to identify and understand the benefits of collaboration as well as the constraints to working together to improve chemicals management. 
• The project exceeded the expected quota (70 per cent) of participants coming from developing countries and countries in transition. The evaluation found that 73 per cent of workshop participants come from developing countries, with 12 per cent being from countries in special situations and 61 per cent from other developing countries as defined by the UN. Only 3.5 per cent of respondents reported having a disability and 13 per cent of respondents were between 18-30 years of age. 
• However, the survey also found that the Toolbox is more relevant to government (48 per cent) and academia (21 per cent) than the private sector (13 per cent).  
• While the IOMC Toolbox website offers a wealth of information on chemical safety, it is not necessarily a one-stop-shop for all such information. The evaluation identified many other reputable sources of information on chemical safety, in particular the PO’s own websites, as well as government agencies, industry associations and academic institutions. These sites can provide additional or more specialized information. 
• One potential weakness of the project is the rather low expected number of visits to the Toolbox portal – the project’s main output – to low hundreds rather than thousands of visits per month. Low expected numbers come from the focus of the Toolbox on the requirements of government staff engaged in sound chemical management, which does not require repeated visits. 
• The articulation and analysis of the project’s ToC helped in reaching the finding that a huge part of project activities – the holding of workshops and webinars – are too short and piecemeal to make visible contribution to better chemical management at the national scale.  
• The Toolbox and its toolkits and documents do not explicitly employ a human-rights approach, nor a gender mainstreaming and inclusiveness strategy.

• While the Toolbox will always require funding to keep it up to date and relevant to evolving needs, the project has developed a plan to sustain the Toolbox after external funding ceases.

Based on the findings and conclusions above, the evaluation produced ten recommendations touching on seven themes. 

Recommendation 1: On gender  

Develop and use an explicit GEEW strategy for the project that builds upon the work of Women and Gender at SAICM. This should include developing a GEEW entry point for the Toolbox. 

Recommendation 2: On the theory of change and targets  

Reflect on the ongoing validity of the Phase III project theory of change at the Phase IV MTE, by filling out a third column added to Table 4 in the main body of this report.  

Review and adjust the baseline and percentage increase per year for targets in the project logical framework, to ensure they are set at a realistic level. 

Recommendation 3: On capacity development  

Develop and implement a capacity development strategy that includes the Kirkpatrick framework and individual, organisational, and enabling environment dimensions of capacity development, as well as guidelines for when to hold in-person meetings and when cheaper virtual meetings will suffice. Build national networks of Toolbox trainers of trainers taking advantage of UNITAR’s experience with capacity development and UNIDO’s experience with ToT.  

Informed by this strategy, Phase IV will be able to better follow up on how Toolbox users and workshop and webinar participants are using project outputs. Success cases should be developed for communication purposes. The success cases should show how the Toolbox has contributed to specific outcome trajectories relating to better chemical management at country level. 

Recommendation 4: On the administrative and financial collaboration 

Allow for staff time and budget to deal with the administrative and bureaucratic impediments identified in Phase III that happen when running a multi-partner project, and which cannot be changed at project level.  

Recommendation 5: On linkages  

Phase IV of the project should take the opportunity to set a good example of POs working together to establish inter- and intra-sectoral partnerships, networks and collaborative mechanisms to share information, experiences, and lessons learned. This could include organizing capacity building workshops jointly, i.e., in a ToT format, where other organisations are invited as co-organisers. 

Recommendation 6: On likelihood of impact 

Embed the project more deeply in national chemical management processes by contributing to carefully selected ones, such as building a cross-sectoral and integrated approach to ensure the sound management of chemicals. In this context, identify and support a network of ‘Toolbox’ champions to increase the number of project beneficiaries at national level. 

Recommendation 7: On likelihood of sustainability 

Phase IV of the project should explore building complementarity between the Toolbox and SAICM's Knowledge portal to sustain the Toolbox after external funding finishes.  

Phase IV should endeavour to make the Toolbox relevant to a broader audience and find ways of making it useful on an on-going basis, so users return to the site. Phase IV should set itself the target of increasing visits to the web site by an order of magnitude to make it more likely to sustain funding to keep it going. 

Three lessons learned were derived: 

Lesson 1: It is important for training of trainers to include at least one module on a trainer's skillset and training methods such as the ADDIE model and making trainer selection in a way that they are likely to train afterwards. 

Lesson 2: It is important to include finance officers in negotiations for new proposals to avoid accounting problems and training on budgeting for projects with multiple partners is key for the successful financial management of a project.  

Lesson 3: Budgetary incentives may be needed to induce partner organisations to work together.