Annex 7 | Tips for appraisers of reports
Annex 7 | Tips for appraisers of reports
Preparing appraisals of reviews, evaluations and research reports
Introduction
The NZAID Evaluation and Research Committee (ERC) Process Guideline provides guidance (page 5) on the appraisal of a review, evaluation or research study. These ‘tips’ add to the NZAID ERC Process Guideline.
The purpose of an appraisal is to:
- provide an assessment of a FINAL report to the NZAID task manager that has commissioned a review, evaluation or research project to support a submission to the ERC
- identify learning and issues for NZAID organisational development
- provide advice on whether to publically release a report (or not)
- raise points for discussion at the ERC meeting.
The appraisal should be concise, and generally no more than 1-2 pages. However, task managers can request additional comment from an appraiser which would be useful to them (e.g. detailed comment on recommendations, and/or the way forward for the activity) which need not be included in the appraisal for the ERC submission.
Aspects to include in the appraisal
In the appraisal include those aspects that are of particular relevance to the report being appraised, will add value to the ERC submission, or are of particular importance to NZAID.
1. Assessment of whether the report meets the Terms of Reference (TOR) (or evaluation or research plan)
If the TOR have asked for an ‘evaluation plan’ or ‘research plan’ (or similar), the appraisal should address whether the report meets any variations and/or additions to the TOR in this plan, as well as whether it meets the TOR.
Where there are deficiencies in the TOR that have affected the quality of the report, these should be noted and may need to be considered, in assessing whether the report met the TOR.
These questions can be asked when assessing whether the report has met the TOR (or evaluation/research plan). Has the review, evaluation or research:
- covered the scope, fulfilled the stated purpose, met the objectives and answered the review/evaluation/research questions?
- been conducted in line with the stated methodological approach (eg was it participatory, build capacity, in partnership?), and the design of methodology as described in the TOR and/or evaluation/research plan?
- included the stated stakeholders or research participants?
- addressed crosscutting issues as described in the TOR or evaluation/research plan?
2. Assessment of the quality of the report against the relevant standards (eg DAC Evaluation Quality Standards and NZAID guidelines and policy)
These questions can be asked to assess whether a review or evaluation is of good quality. Some of these may also be appropriate for research reports.
- Methodology: Is the methodology appropriate (provides valid and reliable data) and clearly described? Has information required to answer evaluation questions/achieve objectives been gathered from appropriate sources? Are data collection, and data analysis methods clearly described? Are ethical considerations described? Are limitations, and their affect, discussed?
- Findings and recommendations: Are findings supported by evidence, free from researcher bias, coherent and logical? Are the sources of findings clear (while maintaining confidentiality)? Do findings answer the evaluation questions and meet the objectives? Are the recommendations and lessons learned relevant, useful and based on the findings?
- Has the report adequately addressed cross-cutting issues, and value for money?
- Does the report show evidence that the review/evaluation addressed the NZAID Evaluation Principles of transparency, independence, partnership, participation and capacity building, or states why they are not addressed?
- Reporting: Has the report adequately described background, context, rationale, purpose and scope of the review/evaluation? Are objectives of the review/evaluation stated? Is the report well written, logically organised and well presented, with a good executive summary and free from typographical errors? Is any confidential material in a separate annex?
3. Implications of the report for NZAID business
Where the appraiser has technical knowledge of the subject of the review, evaluation or research, there is scope in the appraisal to offer expertise. For example, the appraisal may note the implications of conclusions and recommendations, and/or whether conclusions and recommendations are well founded and would align to good practice. The appraisal should highlight anything in the report (and particularly the recommendations) that should be discussed or noted at the ERC meeting as it has particular importance for NZAID’s organisational learning or development practice.
4. Advice on public release of the report
- Is there anything that you believe would prevent the public release of the report (see the report release checklist in the ERC Process Guideline)?
- Provide advice to the person or team who commissioned the review, evaluation or research on whether you believe the report can be publically released to help them to make a recommendation to the ERC on public release.
Example of an appraisal:
Appraisal of Review of XXXX Pacific Support Programme
Overall
Overall this is a good review report that is well written, well presented and well structured.
Does the report meet the TOR?
The TOR for this review noted six objectives for the review (listed under the ‘purpose section’). The TOR also listed a large number of ‘review objectives and questions’ (Annex A of the TOR) which were loosely, but not directly related to the TOR objectives. This lack of direct logic between review questions and objectives made it a little confusing as to what exactly was asked of the reviewer.
However, the reviewer used a sound structure for the report which did not directly relate to either the TOR objectives or the related questions but largely addressed both the objectives, and the questions (Annex A) in the TOR.
The first objective (under ‘purpose’ in the TOR) asked that the review build knowledge and understanding of the project including progress towards achieving objectives and areas of potential impact. It did not directly ask for an assessment of outcomes and impact, and in fact it would have been unrealistic to ask for an impact assessment. The questions in Annex A asked for any emerging unintended outcomes positive and negative. The objectives also asked for a review of the extent to which the goal and objective of the 2007 design had been achieved.
The reviewer addressed progress for each of the project components, and progress towards goal and objective (Section 4.2 Pages 14-20). Constraints around measurement of progress towards outcomes were noted (Page 19). Main outcomes that emerged during data collection and analysis for the review (Page 23) and potential impacts (Page 24), are listed in a separate section.
While the analysis of effectiveness (in terms of achieving outcomes and objectives) would ideally have been addressed in more detail, it did satisfy the TOR. Other review objectives were addressed comprehensively, and the purpose of the review was fulfilled.
The methodology section in the review report and the work plan (Attachment 5), indicate that the review met the TOR in terms of the methodology and approach specified in the TOR.
Did the report meet quality standards?
The report met the DAC evaluation quality standards in most respects. The background and context was good, the purpose of the review and objectives were clearly stated, the report presentation and executive summary were good. The review appears to have been transparent and independent.
The findings were consistently based on clear lines of evidence which added to the reliability of the report. The methodology was reasonable well described. The information sources (documents and people) were clearly linked to the review questions, and constraints were clearly listed. However the report failed to describe how the data that was collected was analysed.
The report was well presented and met the standards in the NZAID Guideline on the Structure of Review and Evaluation reports. The executive summary was good.
Recommendations and implications
Recommendations arise from findings and appear to be useful and relevant.
It is encouraging that the review noted that the programme had made a ‘significant and positive contribution’, however it was disappointing that the contribution could have been more comprehensively described (especially in terms of achieving expected outcomes).
It is clear that the review was constrained by issues around programme logic and monitoring and evaluation (particularly of outcomes). There is important learning in this for NZAID and Recommendation 3 is very pertinent.
Report Release
I see no reasons for withholding this report.