An Independent Evaluation of UNOSAT Rapid Mapping Service (RMS) was published in July 2018. Using a mixed methods approach, it was the first independent evaluation of the Rapid Mapping Service since its inception in 2003. The evaluation included an online survey with a 20.8% response rate and key informant interviews covering 38% of institutional stakeholders, along with a theory of change analysis and a comprehensive desk review. For each evaluation criterion, the evaluation applied a rating, using a four-point scale traffic light system of green, amber and red.as practiced by the United Kingdom’s Independent Commission for Aid Impact.

The most highlighted limitation in the report is the fact it is not always clear who the end user of the service is, for what purpose they are using the maps, and to what ends. This is a common uncertainty among other service providers of satellite imagery, such as MapAction.

The service was found to be fully aligned with the 2014-2017 strategic framework, under programme objective 5. It is also fully aligned with the new 2018-2021 strategic framework, under strategic objective.

Relevance

The evaluation found the relevance of the Service to be very high, with a score of 95%, giving it a “green” score. In four out of five sub-criteria, the service showed a solid performance.

Regarding relevance to the SDGs, a theoretical link is made between the Theory of Change (ToC) and SDG 11.5, and there is evidence for support for SDG 13.1, although this evidence was outside the evaluation period.

The Rapid Mapping Service was found to be relevant for donor and partner needs, and the ToC was shown to be valid.

Efficiency

The evaluation found satisfactory achievement in most areas, but partial achievement in others. The score given was 75%, or amber/green.

For most stakeholders interviewed in OCHA and UNDP no replacement for the Rapid Mapping Service exists, but there are alternatives at the regional level and the European Union's Earth Observation Programme Copernicus is emerging.

Partners like UNOCHA stress the desire for more strategic engagement and dialogue with UNOSAT but understand the restraints on this.

The timeliness of the Service was highlighted, with the Rapid Mapping Team being on call 24 hours a day, 7 days a week.

Effectiveness

The evaluation found that the Rapid Mapping Service shows satisfactory achievement in most areas, but partial achievement in others, again with a 75% score, or amber/green.

As a result of NORAD funding, there are now log frame and results frameworks specifying targets and indicators for the service, although these were not taken into account in the evaluation.

Overall, stakeholders were very happy with the effectiveness of the service. Over 66% of stakeholders found the contribution of the Service to evidence-based decision making as high to very high. For 28% of stakeholders the contribution is medium, and for 2,7% it is low.

49,2% of Service user and institutional stakeholders perceived the contribution of the Rapid Mapping Service to enhance operational coordination as high to very high, followed by 34,9 % medium ratings and 11,1 % low to very low ratings.

5 main factors were identified as affecting the performance of Rapid Mapping Service: Timeliness of maps or other products/services; Product delivered corresponding to user needs; Level of meeting quality expectations; Ease of interpreting the product; Reach of the required distribution channels. All but the last factor were enabling factors.

Impact

The evaluation found that the impact of the Service shows satisfactory achievement in most areas such as the difference made to partners, cumulative effects of the Service and their comparative advantage concerning timeliness and cost. The counterfactual shows however that alternative options to the UNOSAT Rapid Mapping Service are available and could be used as a replacement for most users participating in the evaluation. The score for impact is 71% or amber/green.

The evaluation finds that the actual costs of funding the Rapid Mapping Service for the donor Norway are low compared to the value of the Service to its partners.

The counterfactual of what people would use if the service didn’t exist shows that a large majority (78%) would use another open source service.

Sustainability

The evaluation found that the sustainability shows unsatisfactory achievement in most areas such as financial sustainability, internal operational sustainability or the factors affecting sustainability, with some positive elements such as inter-institutional sustainability through partnerships and the contribution to better humanitarian assistance in the long term. The score for sustainability is 40% or amber/red

Overall picture shows that the results are unlikely to last, unless there is serious change in the business model, financing and staffing of the Rapid Mapping Service

1. UNOSAT should enhance its visibility.
2. Revise partnerships to include more joint planning and implementation tasks.
3. Redefine its business model, with several options proposed.
4. Identify indicators and targets for the outcome and impact of the RMS, in a move to results-based management
5. Assess user-based real-time impact by upgrading existing technical solutions in any future business model.
6. Focus on Risk analysis/possible scenario definition maps and location/preliminary situation maps due to the more direct access to end-users for those products.

• Partnerships are instrumental in delivering results.
• Effective management of documentation, information and data is critical to track progress in highly activity-driven undertakings.
• Evaluability assessments are useful when engaging in a new programming area.

“The quality of the Rapid Mapping Service makes a real difference. There is a UNOSAT standard.”