Assessing the coding reliability of work accidents statistical data: how coders make a difference

Abstract

Introduction: This study assesses the reliability of the coding procedure for a set of variables belonging to the European Statistics of Accidents at Work (ESAW). The work focused on the Portuguese data and experience with the system. In Portugal, this task has been systematically carried out by GEP (the governmental Cabinet for Strategy and Planning), here defined as the “reference group” or “expert group”. However, it is anticipated that this coding task will be performed by non-expert people, since paper-forms will be replaced by e-forms, similarly to what happened in a few EU countries. Objective: This study aims to: 1) assess the current situation, i.e., to quantify reliability of data coded by GEP (reference group), and 2) assess the impact on the reliability level when the coding is carried out by non-experts (two different groups of coders). Methods: The study comprises the estimation of both intercoder and intracoder reliability for a set of 8 nominal variables. The assessment applies 3 reliability coefficients calculated by 3 software packages. Results: The results reveal that the expert group (GEP) holds good to excellent reliability (inter- and intracoder agreements), between 68-98%, while there is a considerable “loss of reliability” (-5% to -39%) when the coding process is transferred to other people, without special training or knowledge in this task. Conclusions: this work gives quantified evidence that reliability of coding accident data is substantially affected by the coders’ profile. Moreover, certain variables, regardless of the coder, systematically hold a higher level of coding reliability than others, suggesting that certain codes may need improvement. Future studies should assess coding quality across the EU countries using the ESAW protocol. Practical Applications: directions for improving the quality of accident data and related statistics; data that is used by researchers and governmental decision-makers to derive prevention strategies.