Liberia: End of Project Evaluator - Liberia

The MENTOR Initiative (MENTOR) is a not-for-profit organization registered in the UK. MENTOR is focused on reducing death and suffering due to malaria and vector borne diseases in emergencies. MENTOR works in 6 countries – Kenya, Liberia, CAR, Chad, Sudan, Angola.

In Liberia, MENTOR with funding from the Comic Relief Special Initiative (CRSI) is implementing a 4-year project under the title “*****Advancing the frontiers of sustainable malaria control in Liberia – A model for Africa*****”. The project started in April 2011 and will run till October 2015. The program focuses on piloting two key innovative malaria control strategies:

1. PSACT (Private Sector ACT) Project: Increasing access to effective diagnosis and treatment for suspected malaria cases through a pilot project working with private medicine stores and pharmacies serving the slum communities of West Point & Bushrod Island in Monrovia – Liberia
2. DL study: Assessing sustainable and effective malaria prevention through the establishment of a large scale pilot on the effectiveness and acceptability of durable wall lining (DL) as a malaria prevention method in Bomi County – Liberia

The programme now is in its final phase and due for end of term evaluation as a part of a donor requirement and this ToR seeks to identify an external consultant who would undertake an independent review of the program.

Evaluation Goal.

The goal of this evaluation is to assess and critically analyse the progress and achievements of the PSACT project and the DL study, the methods used, planned strategies and timeframe to identify assumptions and lessons learned in order to provide a comprehensive evaluation of the CRSI, in accordance to donor requirements (see below section Previous Evaluation).

Evaluation Questions.

The process of the evaluation should answer the following questions regarding the PSACT project and the DL study by using participatory instruments.

Programme Effectiveness.

1. Outputs and objectives:What progress has been made towards achieving the planned outputs and objectives? Specifically towards the outcomes of the grant start up form and the learning questions.
2. Programme Coverage: Has the programme reached all the participants and beneficiaries in the programme’s selection criteria?
3. Programme Participation:Has the programme engaged sufficiently and appropriately with project stakeholders, in particular project participants and beneficiaries?
4. Programme Capacity: Has there been sufficient capacity to implement and manage the project effectively?
5. What difference has theprogramme**made to people’s lives (what, how, who, where, when)?**
6. Who has benefited (by age and gender) and in what ways?
7. Are those changes (outcomes) relevant to people’s health needs?
8. Are they likely to be sustainable in the long term?
9. Have there been changes to policies, practice and attitudes of decision and policy makers to benefit the project’s target groups?
10. To what extent has the project contributed to the achievement of broader malaria national and international policies, conventions, targets etc in Liberia?
11. To what extent has the achievement of the changes/outcomes been influenced by external context and other factors?
12. Supervision, monitoring and data collection: Is the project able to measure project outcomes and indicators with the supervision, monitoring and data collection strategies currently in place?
13. Is data collected in a timely manner?
14. Is data being managed effectively?
15. Are indicators produced valid and reliable?
16. Are progresses towards outcomes properly investigated and taken into account?
17. Sustainability: What are the barriers to sustainability and what recommendations can be made to improve sustainability.

How has the project made this difference?

Approaches used by the project and implementing organization:

• Has the overall theory of change for these projects been effective in starting to bring about lasting change? Were there any gaps?
• What have been the most effective methodologies and approaches the organization used to bring about changes to people’s lives? What has worked and what has not? What lessons have been learned? Who have they been shared with?
• How has the type of organization funded (international NGO) helped or hindered the delivery of lasting change?
• How have relationships between partners throughout the relationship chain (looking at UK organization-local partner(s)-target groups) helped or hindered the delivery of change /outcomes?
• How effective have the project’s management, monitoring, learning and financial systems been? How have they helped or hindered the delivery of lasting change?
• Has the project been cost effective?

Approaches used by Comic Relief:

• How have Comic Relief’s grant making policies and processes (e.g. how we define our programme strategies and outcomes, how we assess applications) helped or hindered the delivery of lasting change?
• How has Comic Relief’s approach to grant management (e.g. individual work with grant holders, and learning activities with other funded organisations) helped or hindered the delivery of lasting change?
• How has the way Comic Relief used its organisational assets helped or hindered the delivery of change (e.g. use of the media, access to decision makers)?
• Are there any other ways in which Comic Relief has helped or hindered the delivery of change?

Project Review and Methodology.

1. It is expected that the evaluator would review and make an analysis of the relevant documents such as project proposals and project reports.
2. It is expected the evaluator will apply a participatory and consultative methodology throughout the assessment, engaging all stakeholders including Comic Relief grant staff and MENTOR HQ staff as follows
3. Interview with the Country Director, Technical Coordinator, DL Coordinator and Principal Investigator, Program staff – IEC BCC Coordinator, PSACT Training Coordinator, PSACT Supply Supervisor, DL Coordinator, DL Entomologist, Data Officer, Logistic Coordinator and Finance Coordinator.
4. Interview with NMCP – Program Manager, PSACT Coordinator, Vector Control Coordinator and other relevant MoHSW staff
5. Interview with other stakeholders – Pharmacy Board of Liberia, Liberia Medical & Health Regulatory Authority, Bomi County health Team, etc.
6. Interview with the beneficiaries – PSACT: Proprietor of MS/P, trained dispenser, client); DL: residents sleeping with DL in their room, village chiefs, CHVs, etc.
7. Discussion with MENTOR HQ staff
8. Interview with Comic Relief Grant Manager

Project Area.

The project area is West Point and Bushrod within Monrovia for the PSACT pilot and Bomi for the DL project. All pre and post field trip write up must be conducted from the evaluators own premises.

Documentation:

The available documents related to The MENTOR Initiative project are as follows:

• Project Proposal.
• Grant Start up form
• Project Reports
• Project Data (quantitative and qualitative)
• Study protocols
• Published papers on relevant areas
• Training reports
• Supply records
• Documents developed by MENTOR – curriculum, IEC materials, presentations, etc.

Previous Evaluation.

A mid-term review of the CRSI has been undertaken in December 2015. The end of project evaluator will take into account this previous report and address the concerns of the donor regarding this mid-term evaluation, which was not validated by CR. Main comments and questions raised were:

• No involvement of the donor in drafting the ToR of the mid-term evaluation, and no interview of the donor grant manager during data collection work
• No interview of MENTOR HQ members conducted during data collection work
• Not all relevant documents were reviewed or formally listed by the evaluator (proposals, annual reports, feedback from CR, study protocols and M&E design)
• Lack of clarity in the evaluation report plan, lack of context regarding the evaluation and background of the grant
• Lack of documented references or explanations from evaluator findings to support some of the report statements
• Major questions were not investigated during the mid-term evaluation: How has this report informed the project? Have the protocols been reviewed? What recommendations were drawn from the evaluation? What are key issues which need to be addressed? Sustainability of the PSACT project (including waste management, pricing, referral of malaria negative patients and Global Fund involvement) were not mentioned.

MENTOR will support the evaluator in addressing these issues by providing all required information prior, during and after the evaluator visit in Liberia. This end of project evaluation should be a fair assessment of the CRSI, in line with the open discussions and critical reports that MENTOR has shared with Comic Relief since the beginning of the grant.

Qualifications of Evaluator.

1. The evaluator should have a minimum of 3 to 5 years of experience of working with health projects preferably with interest in malaria.
2. The evaluator should provide a 3-5 page proposed evaluation plan to undertake this assignment.
3. The evaluator should submit 1 sample of a previous evaluation report.

As a contractor of The MENTOR Initiative the evaluator will be expected to closely adhere to and respect The MENTOR Initiative’s Code of Conduct including sexual harassment guidelines and security regulations.

Evaluation will start 1st week of September 2015.

Timeline and Deliverables.

The final evaluation will be undertaken and completed between October and November 2015. An indicative timeline is given below, but this will be negotiated and finalized once the evaluator has provided his/her own evaluation plan to undertake this assignment.

Desk review of key documents, lit review, upon signing of the contract3 days Travel to Liberia 1 day Meeting with the MENTOR Liberia team – Project familiarization 2 days Meetings with stakeholders – NMCP, PBL, LMHRA, Field visit to enrolled MS/P, interviews with beneficiaries 5 days Travel out of Monrovia 1 days Compile the report 4 days Provide preliminary feedback to the in-country stakeholders 1 day Finalize and submit draft report for review 3 day - Approximately 20 days in total.

Additional working days are likely to be required after feedback has been gathered from MENTOR HQ and Comic Relief.

Reporting.

The evaluator should provide final evaluation report in English, consisting of the following sub headings:

1. Executive summary
2. Introduction and background to the evaluation that is relevant to the report’s analysis and conclusions
3. Methodology used
4. Context analysis that include findings, conclusions and assessment
5. Lesson Learned
6. Specific actionable and prioritized recommendations, considering the future of the projects after end of CR funding
7. Guidance and road map of how the document will be disseminated and findings shared
8. Annexes
9. Terms of reference for the evaluation
10. Profile of the evaluation consultant
11. Evaluation schedule
12. Documents and technical details consulted during the evaluation
13. List of informants
14. Field data used during the evaluation
15. Bibliography.
16. Background information if required

Draft report should be discussed and presented to MENTOR and Comic Relief prior to finalization.

This evaluation is being funded by The MENTOR Initiative.

Please submit your CV along with other documents as described above to recruitment@mentor-initiative.net