An Independent mid-term Evaluation of the Inter-Organization Programme for the Sound Management of Chemicals (IOMC) Toolbox for Decision Making in Chemicals Management – Phase III Project was undertaken between November 2019 and early 2020. It aimed to assess progress towards achieving the project’s planned results and assessed the relevance and effectiveness criteria.

Using a mixed methods approach, the evaluation included an online survey and key informant interviews with stakeholders, along with a theory of change analysis, a comprehensive desk review and three training workshop case studies.

The evaluation had several limitations, the most important being that the changes to the online Toolbox platform had not been completed when the evaluation took place. A second limitation is that the evaluation was conceived from the start as a ‘light’ evaluation without travel to partner countries, although the evaluation team made opportunistic use of the Minamata COP in November 2019 to interview key country stakeholders. Third, as with many evaluations, a considerable amount of the qualitative data collected was based on individual, subjective perceptions and opinions. And fourth, the response rate of the online survey was low (16 per cent), despite two staged reminders.

The evaluation found the project to be relevant with alignment to SDG target 12.4 on the sound management of chemicals, SAICM and SAICM beyond 2020, and the EU’s strategic objectives. The evaluation found that delays in sharing a new version of the Toolbox platform online changed the nature of project workshops from what was planned and that; however, the PMG has not adjusted to make full use of existing quality standards for design and delivery of training related to the inclusion of learning or application objectives in the agenda and no or partial participants’ evaluation. Partly as a result of the delay, Toolbox workshops have provided participants with valuable networking and peer-to-peer learning opportunities that otherwise may not have gained so much prominence.
The analysis of the project’s theory of change shows that the project’s emerging impact pathway is essentially catalytic and nonlinear. This has a number of implications, not least with respect to the choice of indicators, targets and monitoring approach based on use of online questionnaires.

Current levels on use of the Toolbox are ahead of target. However, the evaluation found the targets to be too low to make a significant contribution to SAICM’s objective and there is an urgent need to prioritize efforts to go to ‘scale’. The evaluation further found that the project does not consider gender but should, especially considering the priority that SAICM attaches to it beyond 2020 process. Another key finding of the evaluation includes that administrative and bureaucratic requirements are impeding efficient project delivery and that the project requires a no-cost extension and a fourth phase to stabilize and amplify progress to date, and ensure the Toolbox continues to be maintained after the end of the project.

Based on the above findings, the evaluation issued seven recommendations:

1. The PMG should continue to make finishing the new Toolbox platform its main priority, in addition to prioritizing the development of case studies that include country examples of usage of Participating Organization’s guidance material in tackling chemical management challenges, with lessons learned that can be of relevance for other countries.
2. The PMG should review the project’s theory of change and log frame including the indicators and targets in the log frame, in particular the project’s choice of impact target, consider whether there are key causal processes and assumptions missing from the theory of change and use the theory of change and revised log frame for planning activities for the remainder of the project, including any no-cost extension or fourth phase.
3. The PMG should ensure that peer-to-peer learning on the use of the new online version of the Toolbox is considered in the workshops and should enhance the training guidelines so that training workshops incorporate learning objectives and are informed by the results of training self-evaluations.
4. The PMG should explore ways of increasing project reach and impact, including reciprocal agreements with other chemical-related portals and platforms to point users to the Toolbox platform and encouraging each organization participating in the project, and DG Environment, to stipulate that future chemical-management-related projects include a component in the Toolbox.
5. The PMG should develop and implement a strategy to address women’s empowerment in the Toolbox. The strategy should consider measures suggested by the Women and Gender @ SAICM group in their joint position paper.
6. The PMG should explore ways in which tranche payments can be made in a timelier manner, for example by introducing flexibility to the rule that requires 70 per cent of the previous tranche budget to be spent before the next payment can be made, and in finding ways in which the preparation of the certified consolidated financial reports can be made less onerous.
7. The PMG should continue discussions with the EC and request a no-cost extension of one year and a fourth phase. One requirement for either option is that the project partners agree how the Toolbox website and relevant Toolkits will be maintained after the end of the project. A second requirement is that any extension and new phase is aligned with the beyond 2020 SAICM vision that is currently being developed as part of the intercessional process.

• Planning for collateral outcomes in training events: When people are brought together for a training event, there will always be ‘collateral’ benefit that comes through participants making contact and sharing their ideas and experiences with others. The theory around “boundary objects” can help better understand and plan for these less visible but important knowledge-sharing processes to happen and be maintained.
• Inter-agency collaboration: There are good reasons why UN and other multilateral agencies should work together more closely to achieve the SDGs. In practice inter-agency collaboration is made difficult by bureaucratic requirements established to ensure tight accountability for funds provided.
• Developing and using a theory of change: Including a theory of change in a project document and revisiting it during the mid-term and final evaluations is generally seen as good practice. The lesson is that such a use of theory of change should be considered in other projects.