Evaluation & Research Committee Process
Submissions to the Evaluation and Research Committee
The findings of all reviews, evaluations and research studies commissioned by NZAID must be submitted to the Evaluation and Research Committee.
This ensures that a systematic approach to following through on any findings and review/evaluation recommendations is instituted within the agency, and that the learning from all review, evaluation and research studies is disseminated throughout the agency.
For the same reason, staff are also required to submit to the Committee any reviews and evaluations commissioned by NZAID’s partners that relate to Activities or programmes with significant direct NZAID funding support. This acknowledges NZAID’s partnership approach and the variety of aid modalities in use. Generally, such reviews and evaluations only need to be noted by the Committee; in such cases, the relevant programme area should use its discretion in determining the length of the covering submission. The template for submissions attached to these Guidelines is not always appropriate in those cases.
Research from contestable funds such as the International Development Research Fund (IDRF) and the Post Graduate Field Research Awards (PGFRA) is not submitted. A list of research funded under these schemes is submitted to the Committee for noting at least once a year.
Once the subject and general timing of a submission are known, the staff member who will prepare the submission should contact the Evaluation Team Administrator (or, in their absence, the Evaluation Team Leader) as soon as possible to have the study placed on the agenda for the relevant meeting of the Committee. Agendas often fill up more than a month in advance.
Appraising a Research or Evaluation Study
Appraisal of a review, evaluation or research study is an assessment of the final report.1
Appraisal is conducted by an individual who was not involved in the report’s preparation other than in an advisory capacity. For reviews and evaluations the appraiser/s should have minimal engagement (preferably none) in the Activity or Programme under examination, and in the review or evaluation itself. The following explains the process for appraisal of a review, evaluation or research study:
- At least one person should appraise the final report of each study.
- An appraisal of the final report would normally be conducted by one or more of a SAEG Advisor and/or an external expert.
- Normally, appraisal is undertaken by an NZAID staff member.
- At least one of the appraisers should preferably not have been a peer reviewer.
- In the case of cross-cutting, sectoral or thematic evaluations or research commissioned by SAEG, the reports should be appraised by a person with knowledge and expertise in the relevant area.
- Comments from the appraiser/s should be attached to the submission.
- Once the report has been appraised, the responsible DPM, DPO or NZAID Manager, or the SAEG adviser (with support from the wider appraisal group) will develop a submission to the Committee.
The appraisal should cover the following:
- An assessment of whether the report met the original Terms of Reference and Evaluation Plan or equivalent
- An assessment of the quality of the report against the relevant standards.
- For reviews and evaluations, these are the DAC Evaluation Quality Standards that are included in NZAID’s evaluation guidelines; the principles in the NZAID evaluation policy; and the guideline on Structure of Review and Evaluation Reports.
- For research studies, these are standards of academic rigour and coherence.
- An assessment of the implications of the report for NZAID’s business, such as for programming within a geographic or sectoral area, or more generally for aid and development effectiveness (with a focus on the broad issues within the mandate of the Evaluation and Research Committee)
- Checking the report against the criteria in the Report Release Checklist and advice to the programme team, to help in their recommendation on whether to release or withhold the report (see below).
In some instances, a single appraiser will not be in a position to cover all of the above. In this case, the task manager of the review, evaluation or research study should arrange as many appraisers as are necessary to ensure all requirements for an appraisal are met. In such circumstances, it is good practice to highlight to an individual appraiser, at the time of their tasking, the aspects of the report (eg specialised subject content) on which their views are particularly sought.
As appraisal is of a final report, suggestions for improving the report should not be provided. However, more general, forward-looking constructive feedback for the NZAID staff who commissioned and managed the report, or for the consultants or partners who drafted it, may often be appropriate.
Further tips for appraisers of reports are at Annex 7.
What to Include in a Submission
The submission, including all appendices, should be sent to the Evaluation Team Administrator at least five working days before the meeting.
Submissions consist of a completed submission template with the following supporting material appended:
- Executive Summary of the report
- TOR for the study
- A completed and signed (in hard copy) Report Release Checklist
- All appraisers’ comments.
Discretion should be exercised regarding the size of the submission and the specificity of any recommendations.
Having considered the recommendation of the appraiser/s on releasing or withholding the report, the programme team decides on a recommendation on whether to release the report and completes the checklist that is part of the submission template. If the programme team does not reach a consensus, the leader of the team makes the call. The checklist and an explanation of the overall decision process are part of this guideline (see Decision Process for Release of Review, Evaluation and Research Reports). The recommendation of the programme team on whether to release the report will be considered by the Evaluation and Research Committee.
1 In this context, NZAID’s definition of appraisal needs to be distinguished from peer review. Peer review takes place before the research study or report is finalised and hence is not part of the Evaluation and Research Committee process. Peer review provides critical and constructive comments on a near-final draft, late in the process but when there is still an opportunity to revise the final product (eg the lead consultant’s contract is not yet complete). In the case of reviews or evaluations of individual programme activities the peer reviewers need between them to have sector, country, and evaluation knowledge and skills. External peer reviewers may also be involved in this process if needed. back