Detection of AVPU scale consciousness levels on emergency triage videos: A computer vision-based approach using action recognition and analysis of facial landmarks

Abstract

Assessment of patient consciousness is a key part of emergency triage, enabling rapid identification of patients at risk of deterioration. The AVPU scale (Alert, Verbal, Pain, Unresponsive) is widely used due to its speed and simplicity, but it remains subjective and prone to variability. Recent advances in computer vision and machine learning enable automated video-based approaches that may improve consistency in assessment. In this study, 132 simulated triage videos from 52 volunteers were collected and annotated at frame level for microactions such as examiner–patient interaction, eye state, and pain-related facial expressions. A three-stage pipeline was developed to detect interactions, extract microactions, and classify consciousness (conscious vs unconscious). Pose-based temporal models performed best for interaction recognition, with Long Short-Term Memory (LSTM) and Temporal Convolutional Network (TCN) achieving up to 0.90 accuracy. A Convolutional Neural Network (CNN) achieved 0.92 accuracy for eye-state classification, outperforming a traditional baseline, while pain expression recognition showed moderate performance (F1-score of 0.54–0.60). The final TCN-based model achieved 0.92 accuracy for overall consciousness classification. Overall, the results show that microaction-driven video analysis is a promising approach to support automated consciousness assessment in simulated triage settings, while further validation in larger and real-world clinical datasets is required.