9. Critical appraisal
The session will introduce what ‘critical appraisal’ means (i.e. the assessment of how reliable research evidence is) and how researchers can weight evidence according to its reliability in any synthesis. Critical appraisal will be described in terms of the internal validity (relevance) and external validity (generalisability). Approaches to critical appraisal will be introduced and, in a practical session, attendees will experience issues relating to critical appraisal and the level of detail required for the process. This session includes a practical exercise involving critical appraisal of two articles from a recent review.
Learning objectives:
- To understand why critical appraisal is a vital part of all systematic reviews
- To understand the meaning of internal and external validity
- To appreciate practical ways in which critical appraisal can be undertaken and to understand the need for careful planning before assessments begin
- To understand the need for consistency checking between reviewers in critical appraisal
- To understand that the outputs of critical appraisal must be used within a review, either to exclude or weight evidence of different reliability
- To understand ways in which the outputs of critical appraisal can be used in a synthesis
To begin, watch the following presentation:
You can find the lecture handouts here.
Next, read the 8th chapter of the guidance from the Collaboration for Environmental Evidence here. We'll revisit sections of the CEE Guidance throughout the course. It focuses on environmental and conservation topics, but is subject agnostic, meaning that it's useful for other disciplines, too. Feel free to focus instead on the Cochrane Guidance if you work with healthcare topics.
Practical exercise
In this exercise, you will spend some time reading and critiquing the methods used in two primary research articles for a real review that focused on the impacts of terrestrial protected areas on human wellbeing. You can check out the final review here
.- Download a copy of the critical appraisal form - click this link to download a copy of the form on your own computer.
- Download the full texts of the following two articles:
- Conduct critical appraisal - Read through the two articles (focus on the objectives, methods and results sections) and try to fill in the critical appraisal form for each as much as you can. Don't worry if you find it particularly challenging - in reality, you would spend time ensuring everyone knew how to use the form in your review team and that everyone interpreted things as similarly as possible.
Once you have done as much as you can, compare your appraisal with the model answers here.
How did you get on? Was there anything you missed, or perhaps that you disagree on? Some disagreements are to be expected, since this is possibly the most subjective part of the systematic review process. However, by having an a priori approach and by working to ensure consistency between review team members before starting in ernest, a good quality critical appraisal aims to reduce subjectivity to a minimum. By reporting and justifying all processes, decisions and outcomes in as much detail as possible in the final review and its supplementary information, review authors aim to maximise transparency where some subjectivity remains.
Once critical appraisal has been carried out for all studies and the evidence base as a whole, we need to use these assessments in the synthesis. In the next two modules, we will discuss synthesis in more detail.
Move to the next module!