Final Evaluation of the IOMC Toolbox for Decision Making in Chemicals Management – Phase II

The Inter-Organization Programme for the Sound Management of Chemicals (IOMC) brings together nine UN and multilateral organizations that aim to strengthen international cooperation on chemical safety and improve coordination of their own chemicals-related policies and activities through the IOMC.

The IOMC Toolbox for Decision Making in Chemicals Management Project is a logical response to increasing needs and demands from policymakers working within developing and transitional economies. The toolbox was developed to provide an easy, web-based access to a consolidated library of guidance material and other resources, and to provide a structured decision-making tool which would guide users towards the most appropriate, cost-effective chemicals management solutions in line with their own national resource constraints. The toolbox aims to support implementation of Strategic Approach to International Chemicals Management (SAICM). Phase ll aimed to undertake in-depth piloting of the toolbox, develop and integrate considerably more content within the system and undertake a significant programme of promotion and training.

IOMC Participating Organizations

  • Food and Agriculture Organization (FAO)
  • International Labour Organization (ILO)
  • United Nations Development Programme (UNDP)
  • UN Environment
  • United Nations Industrial Development Organization (UNIDO)
  • United Nations Institute for Training and Research (UNITAR)
  • World Health Organization (WHO)
  • World Bank
  • Organisation for Economic Co-operation and Development (OECD)

Toolbox content:

SCHEMES (Phase l and ll)

  • Setting up a Pollutant Release and Transfer Register (PRTR)
  • Industrial Chemicals Management
  • Implementing a Classification and Labelling System for Chemicals (GHS
  • Public Health Management of Chemicals
  • A national management scheme for pesticides
  • An occupational health and safety system
  • A chemical accidents prevention, preparedness, and response system for major hazards

TOOLKITS

  • OECD Environmental Risk Assessment Toolkit
  • WHO Human Health Risk Assessment Toolkit
  • FAO Toolkit for Pesticides Registration Decision Making
  • UNIDO Toolkit on Chemical Leasing
  • UNIDO Toolkit on Innovative, Safe and Resource Efficient

EVALUATION’S KEY FINDINGS

 

+

-

Relevance

  • Toolbox concept was relevant at a global level
  • Highly relevant to the chemicals management-related needs of policymakers working in transitional and developing economies
     
  • Toolbox only directly relevant to a
    very limited audience (national policymakers)
  • Relevant audiences not reached in all target countries
  • Relevance not as clear for non-governmental audiences            

Efficiency

  • Rigorous development of quality-assured material and guidance on chemicals management
  • Delivery of an extensive amount of quality material
  • Inadequate demonstration of the cost-effectiveness of the promotional strategy
  • Flaws of the project’s log frame
  • Poor indicators
  • Weak monitoring processes
  • Hugely delayed report submissions from individual POs

Effectiveness

  • Most outputs delivered or on track for delivery
  • Progress towards the project’s overall objective
  • Externally hosted toolkits are uniformly popular and effective
  • Toolbox’s unpopular platform and interface
  • Promotion and training strategy not entirely effective or appropriate
  • The toolbox is not an effective mechanism for accessing information
  • Reputational risk for the IOMC

Impact

  • Direct and substantive influence on chemicals management policy, legislation and/or practice in at least three countries (Colombia, Myanmar and Zambia)
  • No broader impacts such as long-term social, health or economic outcomes (but too early to expect such impacts)
  • Degree of collaboration between POs could have been stronger  

Sustainability

  • Direct influence of the project strategy on the extent to which the toolbox continues to be used and can at least influence longer-term results
  • Longer-term results of improved chemicals management beyond the direct control of the project
  • No long-term strategy developed for post-project hosting and/or maintenance
  • Significant unmet demand amongst toolbox users for softer, offline support

RECOMMENDATIONS 

The seven following recommendations that aim to strengthen the project have been identified, based on detailed feedback from target users and on the evaluation’s own findings:            

  1. The Project Management Group (PMG) should commission a usability and options study with comprehensive knowledge and knowledge management solutions from a specialist provider. (The usability part should identify core user requirements and assess the technical strengths and weaknesses of the current system). 
  2. The system should at least fulfil the following technical requirements : standard web navigation functions (transparent navigation structure, comprehensive site search, back-button, overall cosmetic design following web design norms and standard web grammar), comprehensive, consistent and continuously monitored translation (from English into at least French and Spanish), minimized number of click-throughs to any content, automatic link auditing, increased cross-referencing and cross-linking between schemes, development of a long-term hosting strategy for the toolbox and its supported toolkits.
  3. Increase the relevance and accessibility to a far broader audience by developing a broader typology and system of content “tagging” allowing users to filter and/or identify materials according to non-technical categories. (The new content categories/tags recommended are : audience and SAICM elements as a minimum).
  4. Avoid high-profile promotion (particularly to large, multi-disciplinary audiences) until the toolbox’s technical flaws have been resolved or a new system has been adopted.
  5. The PMG should develop an alternative training strategy that is explicitly and possibly exclusively focused on engaging and supporting the primary target audience of policymakers (training activities based on detailed stakeholder and needs analysis, ensuring that any workshop bring together all relevant institutions within the target country/region, including formal post-event support and allowing participants to access ongoing tailored advice).
  6. Achieve a self-sustaining network in the long term by the PMG’s initiation of a toolbox-centered, global community-of-practice. (Alternative communication channels should be explored).
  7. The PMG should: reformulate the results framework that should at least identify intermediate outcomes that can directly or plausibly be influenced by project interventions, revise monitoring strategy prioritizing outcome rather than output measurement and revise indicators accordingly.

LESSONS LEARNED

The evaluation identified the six following generalizable lessons which are valuable learning points for the IOMC POs and for all project stakeholders:               

  1. Reaching users at exactly the right time in their policy development cycle is critical: the most important driver of substantive toolbox usage is whether a user has an immediate, tangible policy-related objective or problem to address.
  2. The quantity of toolbox users is not as important as the profile of the users: if the right user accesses the toolbox at the right time, that user’s work could have great impact.
  3. National dissemination and uptake is likely to be more effective when led by a high-level individual or institutional champion with knowledge of ministerial mandates and potential opportunities for cross-sectoral collaborations.
  4. Workshops are more likely to support longer-term impacts if they are targeted at – and directly tailored towards – a highly specific audience with a direct and well-defined motivation to use the toolbox.
  5. It is highly probable that countries graduating from the toolbox will continue to require a degree of technical support. The broader project would benefit from a consistent strategy for providing support - or at least signposting - to countries whose technical needs have outstripped the content of the toolbox.
  6. Neglecting to define and track clear outcome-level results can lead to project strategy being locked-in to delivery of outputs that may no longer be appropriate. Outputs should always be conceived against the actual changes and transformations that the project is aiming to bring about. 

Downloads