New Zealand Aid Tools Activity Tools 

Annex 2 | Examples of Good Practice

Annex 2 | Examples of Good Practice

The following are examples (section by section) of good practice that were included in NZAID TOR for evaluations and reviews.

Purpose and rationale of review or evaluation

Adapted from the TOR for the Evaluation of NZAID support to World Vision - Humanitarian / livelihood response to Manam Island Internally Displaced (2008), this example is well written and includes aspects considered in these guidelines to be important. It specifies the purposes of the evaluation for the different stakeholders

The two World Vision projects which were implemented concurrently 2005-2006 are now complete and it is an appropriate time to undertake an evaluation. The broad purpose of the evaluation is to provide:

  • analysis and recommendations for the Government of PNG which will assist the government to develop lasting solutions to the specific humanitarian and livelihood needs of the displaced Manam Islanders
  • information for NZAID to determine whether and how it should best support solutions that are developed by the Government of PNG
  • general learning from the Manam Islanders situation and the humanitarian/livelihood response as a case study of permanent environmental displacement

Scope

Adapted from the TOR for the VSA Review (2008), this example is well written and covers most of the important aspects that need to be considered in the scope. However, note that scope should ALSO include which of the DAC criteria (efficiency, effectiveness, relevance, sustainability and impact) the review will address.

The time period the review will cover is the duration of the current funding arrangement, that is, 1 July 2003 - 30 July 2007.

The review will consider all VSA’s work except that of the External Relations Unit (public awareness, development education and fundraising). It will not involve undertaking a field visit to Africa, but will be able to draw on the feedback provided by VSA’s partners to NZAID as part of the ongoing Africa Strategy Review. The review will consider key questions in particular detail in two geographical localities.

The main stakeholders involved in the review are staff of both NZAID and VSA, including New Zealand and in-country staff, and in-country partner organisations.

Back to top

Objectives

Adapted from the TOR for the Evaluation of the NZAID/Tonga Graduate Diploma in Public Sector Management Courses (2008), these objectives are well written with clear evaluation objectives associated with each objective. The objectives address the DAC Criteria of effectiveness and relevance.

1 | To assess the effectiveness of the Dip PSM programme in achieving the objectives and outcomes expressed in the original letter of commission with the Massey University Institute of Executive Development (25 March 1998).

  • How effective was the course documentation, teaching resources, delivery mechanisms and student materials for the Dip PSM in Tonga?
  • To what extent did all reports and documentation relate to the various intakes of the diploma programme?
  • Using the data from a tracer study of the graduates, to what extent are the objectives and outcomes of the course reflected in their ongoing work?
  • What are the issues that supported or constrained graduates from implementing the outcomes of the course in their ongoing work?

2 | To assess the relevance of the Dip PSM to the needs of the public service in Tonga.

  • To what extent do the Tongan authorities feel the Dip PSM supported activities of the GOT to strengthen its public sector?
  • Was the material studied under the Dip PSM programme relevant to the management of the public sector in Tonga?
  • How could training assistance better coordinate with and support continuing development and increasing effectiveness of the management of the public sector in Tonga?
  • What other donor activity strengthens the public sector in Tonga, especially with reference to training and capacity building?

3 | To undertake an evaluation of the management of the Tonga Dip PSM programme.

  • Was the reporting satisfactory and meet agreed objectives?
  • Was diploma programme monitored and evaluated effectively during the activity?
  • Was the diploma programme cost effective?
  • Was administration and management of the Dip PSM - including any roles undertaken by NZAID (the DPM and the Post) - and the GOT and Massey University -effective?

Back to top

Value for money

Adapted from the review of the Tongan National Centre for Women and Children (2008), this statement clearly described expectations in terms of reviewing value for money.

While an assessment of relevance and sustainability is the primary focus of the review, NZAID also emphasises a ‘value for money’ approach. The review will therefore provide an assessment of how the money spent compares with the broad outcomes, impacts and changes brought about by the scheme. The review will consider whether savings could have been made without compromising outcomes, eg through improved design or management of the activity.

Methodology

Adapted from the TOR for the Review of the Cook Islands Marine Resources Institutional Strengthening (CIMRIS) Programme (2008), this methodology (although not including all the aspects suggested in these guidelines) clearly described approach without being prescriptive and requested an evaluation plan (termed a ‘design and plan for the review’).

The review will … a participatory approach. This will include (at a minimum) a pre-review briefing with stakeholders (in this case PCC members), participatory information gathering, and an end of visit debriefing to check that the draft findings and recommendations are representative of the stakeholder views.

The team leader will develop a draft design and plan for the review methodology based on the  briefing with CIMRIS Programme Steering Committee in Rarotonga and in consultation with the team member(s) (by telephone and/or email if necessary). The Steering Committee will have the opportunity to review the design prior to finalisation. 

The approach taken will ensure that the views of all PCC members are heard and incorporated into the findings or the review. It will specify the roles and responsibilities of each team member. The draft design of methodology and timeframe for the review will be submitted to the Steering Committee and refined as necessary.

It is not anticipated that a visit to the outer Cook Islands will be feasible within the timeframe. Pearl farmers (from Manihiki) and other people involved in the marine sector in the outer islands and other leading sector members may be contacted by telephone and email of face to face interviews are not possible. Industry representatives are mostly based in Rarotonga.

Cook Islands the review team will:

  1. Attend an in-country briefing with NZAID and the Steering Committee
  2. Conduct the review
  3. Present a draft summary of findings at the end of the in-country mission in the form of a short briefing paper and to a workshop with MMR and other stakeholders.

Management and governance of the review or evaluation

This example from the VSA Review (2008) clearly noted who was commissioning the review, and the composition and function of a steering committee. However, note that this section should also include a note on the management of the review (and who will approve the final report).

NZAID is commissioning the review. NZAID will choose a consultant or team of consultants to undertake the review, in consultation with VSA. A steering group will be established by NZAID  including a representative of SAEG, the Civil Society Programme Manager, the CEO of VSA and another person nominated by VSA.  The steering group will:

  • Provide feedback to the consultant on the review plan and budget
  • Provide practical support to the consultant(s)
  • Provide written comment on a draft report.

Back to top

Composition of the evaluation or review team

From the TOR for the Evaluation of NZAID support to World Vision - Humanitarian/livelihood response to Manam Island Internally Displaced (2008), this section on composition of the review team, and responsibilities, was clear and well written.

The evaluation team will be recruited locally by NZAID in PNG. The evaluation team will have strong professional skills and experience in:

  1. participatory evaluation
  2. humanitarian relief
  3. livelihoods under displacement situations
  4. resettlement within a PNG context
  5. gender analysis
  6. fluency in Tok Pisin and if possible Manam language.

A Team Leader will be appointed by NZAID to lead the evaluation and take responsibility for ensuring the timing and quality of process and outputs and drawing together the final report.  The Team Leader is required to have skills in:

  1. team leadership
  2. excellent communication skills (including report writing)
  3. evaluation
  4. humanitarian and/or Livelihoods sectors.

Two team members - one from OCHA and one from Bismarck Ramu Group have been recruited to ensure full coverage of the above skills.  The team leader will agree on the teams’ respective roles once the team meets in PNG (or in correspondence prior to meeting).  The team leader will be responsible for the final report but the other team members will work on various aspects of the report (as agreed by the team and NZAID).

The Evaluation team leader and members will be sufficiently independent of the agencies involved in funding and delivery of the programme and the affected communities to be able to elicit.

Outputs and reporting requirements

Adapted from the review of the Tongan National Centre for Women and Children (2008), while this example was chosen because it was concise and well written, it should have also have included a description of what would happen to the final report in terms of peer review and appraisal, that the report should comply with the DAC Evaluation Quality Standards, and that a summary of the report will be placed on the NZAID website, and the full report provided if requested (any information that could prevent the release of the report should be included in a confidential annex). Also dates of deadlines for the outputs are needed.

The consultant will produce the following outputs:

Output 1: a methodology framework (evaluation plan) for discussion with NZAID at the outset of the review;

Output 2: end-of-visit workshop in Tonga to feedback draft findings to key stakeholders;

Output 3: a verbal debrief with NZAID (both New Zealand High Commission and NZAID Wellington staff, by teleconference) following the completion of the end-of-visit workshop;

Output 4: a draft written report within five days on completing the in-country review;

Output 5: a final written report within five days of receiving feedback from NZAID.

The consultant should refer to the NZAID guideline on the structure of review and evaluation reports to ensure correct formatting and structure. The Contractor shall ensure that the assignment is carried out with due diligence, efficiency and economy in accordance with the time specified in this Contract, observing sound management and technical practices, and complying with professional consulting standards.

The Tonga Programme will prepare a submission to NZAID’s Evaluation Committee. This Committee will make recommendations regarding the distribution of the report within NZAID and advise whether any additional external distribution steps need to be taken. 

Back to top

Follow up of evaluation or review

This example from the review of the Tongan National Centre for Women and Children (2008) is concise and well written and describes what will happen after the final report has been accepted. It does not, however, consider follow up on preliminary findings.

Following submission of the final report, NZAID (including the Pacific Group, Strategy Advisory and Evaluation Group and the NZHC) will consider the draft report in consultation with the NCWC.  NZAID will then work with the NCWC on implementing agreed recommendations.

Sources of written information

From review of Fair Trade Australia and New Zealand (2008)

  • Oxfam/NZAID Funding Arrangement
  • FTAANZ & FLANZ Joint Business Plan “Moving Fair Trade Forward in ANZ” 2005-2008
  • Programme Logic Framework
  • 7 x Steering Committee Meetings - Papers, Reports and Minutes
  • 4 x 6 monthly Reports to NZAID
  • End of Project Report
  • FTAANZ and FLANZ Project Staff Job Descriptions
  • FTAANZ Constitution
  • FLANZ Constitution
  • FTAANZ/FLANZ Service Agreement
  • FTAANZ Executive Committee Meetings - Papers, Reports and Minutes
  • FLANZ Board Meetings - Papers, Reports and Minutes
  • NZAID Evaluation Policy Statement
  • DAC Evaluation Quality Standards
  • NZAID Guideline on the Structure of Review and Evaluation Reports

Back to top