Skip to main content

Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Risk of Bias Tool in physical therapy trials

Date and Location




Saturday 21 September 2013 - 10:30 - 12:00


Presenting author and contact person

Presenting author

Susan Armijo-Olivo

Contact person

Susan Armijo-Olivo
Maria Ospina
Abstract text
Background: The Risk of Bias (RoB) tool is an emerging method for determining the risk of bias of health trials. Recent research have recommended further testing of the psychometric properties of the RoB domains, and validating the tool in a wider range of research fields. Inter-rater reliability of the RoB tool has been evaluated for child trials, general medical and nursing trials. The inter-rater reliability of the RoB has not been evaluated for physical therapy (PT) trials. Objectives: To test the inter-rater reliability of the RoB tool applied to PT trials by comparing ratings from Cochrane review authors and blinded external reviewers. Methods: Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments of PT trials included in the meta-analyses were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Consensus ratings between the 2 reviewers were compared with the Cochrane RoB ratings. Agreement between Cochrane and blinded external reviewers for individual domains and the final rating of the RoB tool was assessed using weighted kappa (K) for categorical data (K=0.0 to 0.40 poor; K=0.41 to 0.60 moderate; K=0.61 to 0.80 substantial). Results: In total, 109 trials included in 17 Cochrane reviews were assessed. Interrater agreement on the overall RoB rating was poor (K=0.07). Agreement on individual domains of the RoB tool was poor (median K=0.20) ranging from K=0.01 (“Other bias”) to K=0.63 (“Sequence generation”). Conclusions: Risk of bias assessments are not consistent across different research groups. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to apply the RoB tool and revisions to the tool for different health areas are needed.