Remote sensing technologies for physiotherapy assessment

The paper presents a set of remote, unobtrusive sensing technologies that can be used in upper and lower limbs rehabilitation monitoring. The advantages of using sensors based on microwave Doppler radar or infrared technologies for physiotherapy assessment are discussed. These technologies allow motion sensing at distance from monitored subject, reducing thus the discomfort produced by some wearable technologies for limbs movement assessment. The microwave radar that may be easily hidden into environment by non-metallic parts allows remote sensing of human motion, providing information on user movements characteristics and patterns. The infrared technologies - infrared LEDs from Leap-Motion, infrared laser from Kinect depth sensor, and infrared thermography can be used for different movements' parameters evaluation. Visible for users, Leap-motion and Kinect sensors assure higher accuracy on body parts movements' detection at low computation load. These technologies are commonly used for virtual reality (VR) and augmented reality (AR) scenarios, in which the user motion patterns and the muscular activity might be analyzed. Thermography can be employed to evaluate the muscular loading. Muscular activity during movements training in physiotherapy can be estimated through skin temperature measurement before and after physical training. Issues related to the considered remote sensing technologies such as VR serious game for motor rehabilitation, signal processing and experimental results associated with microwave radar, infrared sensors and thermography for physiotherapy sensing are included in the paper.


INTRODUCTION
Stroke is a common health care problem globally, and it is a leading cause of acquired disability worldwide. In 2013, worldwide prevalence of stroke was 25.7 million, with 10.3 million people having a first stroke [1]. The majority of patients with stroke are experiencing incomplete recovery of motor deficits despite intensive rehabilitation, with up to 60% having impaired manual dexterity, 6 months following the stroke. To recover the mobility in order to regain the independence with functional activities intensive physical rehabilitation program is required. Traditional physical therapy can be characterized as hyper-focused workout where the goal is to work the muscles of the affected areas. Traditional physiotherapy incorporates a variety of stretching, strengthening, aerobic training, and pain-relief exercises [2]. The process is long and often painful. The outcome evaluation is usually carried out in subjective way [3] that may difficult the prediction of the required training period for motor performance improvement. The new developments in the field of sensorized equipment for physical therapy [4] [5] made possible the physical rehabilitation session records and reports generation that can be useful for the implementation of objective evaluation of patient evolution during the rehabilitation period.
The development of virtual reality (VR) and augmented reality (AR) scenarios tailored for physical rehabilitation purposes, providing natural interaction for the users based on Kinect or Leap Motion, represents an important research field in the physical rehabilitation technologies for designing strategies that engage patient in their treatment and increase flexibility of training process Several implementation of VR serious games for rehabilitation characterized by the usage of Kinect natural user interface are reported in the literature [6] [7] [8] [9] [10]. In these games the upper limb angles and velocities are calculated using the values of skeleton's joints coordinates (XJi,YJi,ZJi) acquired by Kinect during the serious game.
Leap Motion may be also a solution for dexterity and hands movement training that improves manual precision in patients having essential tremor [11] or those affected by a stroke event [12]. Accurate detection of finger motion during manual activity possible by Leap Motion [13] can be used to detect finger trajectories for an imposed motor rehabilitation training task as referred in [14] or as natural user interface for serious games [15] that was developed by our group.
Both sensors are infrared (IR) optical based sensors and present the advantages of accurate remote sensing of body motion. However, in some cases the detection of whole body (Kinect sensor) or of the hand fingers (Leap Motion) can be affected by the presence of obstacles between the sensor and user under test, wrong localization of the user in relation the sensor, infrared interferences and target texture [16].
In virtual environments, in which the real world is replaced by imaginary objects the localization of the user in the virtual scenario is not critical, while in the augmented reality case the mixt between the real object and virtual objects and the necessity to transmit to user of the AR scenario, the idea of the coexistence of both type of objects requires high tracking quality of the real objects. Good localization would allow the augmentation of the training scenario with virtual objects. The relation between user under rehabilitation process, real objects and virtual objects interaction can be investigated taking into account the physical therapy objectives, the required motor activities and user physiology (e.g., heart rate, heart rate variability, respiratory rate). The remote sensing of cardio-respiratory and motor activity using the Microwave Doppler Radar Sensor (MDRS) allows remote-sensing of human large amplitude motion related to physical rehabilitation plan and activities of daily living (ADL) but also short amplitude motion associated with respiratory and cardiac activity without the need for sensors to be worn by the subject [17][18] [19] which can be important advantage when the user is involved in rehabilitation in real, VR or AR scenarios [20]. Based on the MDRS the gait pattern during the rehabilitation using walking aids can be extracted [21]. The effectiveness of human motion classification based on radar signals has been demonstrated [22] [23].
The evaluation of physical therapy effectiveness is usually expressed by capability of the user to perform upper limb and lower limb movements in VR and AR scenarios as results of intensive muscles activity. The muscle activity during different type of training requires the usage wearable EMG system with wet or dry electrodes. A promissory technology that permits to perform the unobtrusive body temperature variation measurement caused by muscle activity during the physical training is the thermography. It is particularly suitable to precisely map the cutaneous temperature distribution and its evolution during exercise [15] [24]. The thermography may be used for the evaluation of physical therapy effectiveness when user performs upper limb and lower limb movements for muscles training in real, VR or AR scenarios.
The paper is organized as follows: section two presents two natural user interfaces commonly used in serious game for physical rehabilitation and a set of developed serious games; section three presents the microwave Doppler radar sensors for remote sensing of motor activity and cardio-respiratory assessment; section four presents thermography as a method to measure the skin temperature associated with muscular activity during physical rehabilitation; fifth section includes experimental results during the application of described sensing technologies and conclusions are presented in the last section.

II. NATURAL USER INTERFACES
As the worldwide most popular natural user interfaces for physical rehabilitation by serious game can be mentioned the Kinect Sensor followed by Leap Motion [25] [26][27] [28]. Both sensors are based on IR projector (Laser or Led) and IR detectors [29] [30]. The Kinect allows detection and recognition at distance (1-5 m) of human body parts such as upper limb and lower limbs, while the Leap motion permits the accurate detection of the hand gesture and fingers motion, at lower distance (approximately 30 cm). A short description of the Kinect and Leap motion and a set of implemented and tested serious games for physical rehabilitation are following presented.

A. Kinect Sensor
Microsoft Kinect sensor that exist today in two versions (K.V1 and K.V2) allows the acquisition of RGB, IR and depth image with high frame rate (30FPS). An RGB camera that stores three channel data in a 1280x1024 pixels (Kinect v1) or 1920x1080 pixels (Kinect V2) resolution. An infrared (IR) emitter and an IR depth camera (640x480 pixels for K V1 and 512x424 K. V2) allow extraction the depth information. The IR laser emitter deliver infrared light beams and the depth sensor reads the IR beams reflected back to the sensor. The depth measurements are performed using speckle pattern technology [31]. The values of minimum and maximum depth distance varies from K.V1 to K.V2 thus maximum depth distance is 4m in the case of K.V1 and 4.5m in the case of K.V2 while the minimum depth distance is 0.8 m in the case of K.V1 and 0.5 in the case of K.V2. Important parameter that characterize the Kinect is the Field of View (FOV) (Fig.1) Fig.1. Kinect Field of View characteristics a) relation between FOV and the Kinect-user distance; b) K.V1 and K.V2 comparison regarding FOV K V2 presents better depth measurement accuracy the K V2. According with Pagliari et al. [32], for small distances, up to 1.5 m, between the Kinect camera and the target (e.g. user body that performs the rehabilitation) the absolute errors associated with the depth measurement using K.V1 and K.V2 are about 1.5 cm while for big distances between the Kinect and target the depth absolute errors are about 10 cm for K.V1 and less than 2cm for K V2.

B. Leap Motion
The Leap Motion Controller (see Fig. 2) presents optical components expressed by two cameras and three infrared LEDs. This device can track the motion of both hands and all 10 fingers with up to 1/100 mm accuracy and no visible latency within its field of vision. The Leap Motion Controller's viewing range is 60 cm above the device and can reach 80 cm with the Orion beta software [33]  This remote sensing technology holds a great promise for the rehabilitation field, since it does not require the patient to wear any sensing device (e.g. gloves with force sensors [34]). The device provides a new way of interaction between the user and the computer but also provide useful information regarding motor coordination. Hand dexterity in patients with Parkinson's disease can be evaluated imposing graphical tasks. A LabVIEW software for upper limb function monitoring during the execution of graphical tasks was developed by the author. Thus, the user finger pointing different templates (e.g circle, zigzag or Archimedean spiral), finger that is detected by the Leap Motion. At the end of the test the deviation of finger pointed trajectory and the imposed template as so as metrics as finger velocity are calculated.
The tests with IR natural user interfaces to extract upper limb functionalities are very useful on user motor capability evaluation phase, however to motivate the long term training the VR and AR serious game represents the best solution.

C. Kinect Serious Game
Using K. V1 and K.V2 different serious games for upper limb rehabilitation purpose were developed using the Microsoft Kinect SDK and Unity 3D game engine for windows. Thus, games such as Therasoup game [7], AppleHarvesting [8], JustPhysioKidding [9] where designed and implemented. The game are related to the simulation a daily life activities, such as cooking in an indoor scenario. During the game, the player controls the avatar through the Kinect sensor and tries to pick the ingredients landed in shelves and to put them in a pan at the center, such as it can be observed in Fig. 3.
The VR interface designed for the user under rehabilitation presents information about the reached score that corresponds with the vegetables that was picked and delivered to the pan. Considering the rehabilitation purpose, objective

D. Leap Motion Serious Game
The game "Collect Color Cube" was developed by the TailorPhy project's team, using Unity and C# scripts. The interface between the user and VR scenario is based on Leap Motion controller connected by a computer that runs the VR serious game. The Leap Motion captures the movements of the user's hands and fingers, and represents them as part of the virtual game scenario for physical rehabilitation (Fig. 4). In the implemented serious game the goal is to place the colored cubes in the collecting box that presents the same color as the captured cubes in order to earn points. It has a certain established time and the boxes will change the color during the game. A common setup was 4s to 10s the period between color changes. At the same time the physiotherapist can impose the serious game duration according to the capabilities of the user under training. The game has to be adaptable to the player needs to take better advantage of the gaming experience. At the beginning of each game are set some parameters such as: Name, Age, Gender, Hands' Training Selection (left, right, both), Time interval between successive games, Game Time Duration. Depending on the parameters set, the game will adapt to the player, for example, along with Gender and Age will define how the hand will be represented during the game.

III. MICROWAVE DOPPLER RADAR
The motion detection and cardio-respiratory monitoring in unobtrusive way based on microwave Doppler Radar (MDR) technology was considered. As advantages of the microwave radar motion sensors can be mentioned the detection of radial motions in accurate mode independent by environment conditions (e.g. temperature variation, dust). The MDR presents the advantage of penetration through non-metallic objects which permits to cover MDR, making the motion monitoring system invisible for the monitored user. Parameters such as velocity, direction, distance can be measured in remote way avoiding the disturbance of the user during physical rehabilitation sessions. Detecting small amplitude motion the cardio-respiratory condition can be also extracted before and after the training using MDR, the higher amplitude motion artefacts during training making the cardiac and respiration activity difficult to be detected. Two FMCW MDR were used by our team for remote sensing one of them (IVS-162) was used in different applications in the field of cardio-respiratory and motor activity monitoring [35][36][37] [38]. The MDR sensor, IVS-162 (presented in Fig. 5) is characterized by FMCW and FSK functioning modes for a transmission frequency of 24.125GHz. For the used FMCW functioning mode different types of modulation patterns such sawtooth, triangular modulation, can be considered according with measurement purpose. As it can be observed in Fig. 5 MDR presents TX and RX integrated patch-antennas and an advanced PHEMT-oscillator with low current consumption and dual channel operation for direction of motion identification. The FMCW MDR can change its operating frequency during measurement, thus the transmission signal can be modulated in frequency or in phase. This kind of radar permit to measure the distance between the radar antennas and the target by comparing the frequency of the received signal to a reference range of the target accurately. The radar is able to measure very small ranges variations to the target that can be comparable with transmitted wavelength. The relation applied to calculate the distance between the motion sensors and target (parts of the user body under motion during the rehabilitation session) is: (1) Where: T-sawtooth signal frequency, fD differential frequency, Δf -frequency variation of the transmitter oscillator, c0 -speed of light (Fig. 6)   Fig. 6. Transmitted and received waves for FMCW MDR Several tests were carried out using IVS-162. The I and Q signals were acquired and processed in order to extract the time frequency evolution during the target motion. The spectrograms were calculated using the STFT and/or Gabor Transform (GT). According with the target low frequency motion Gabor Transform provide information about the relation between the motion and the time-frequency spectrogram evolution taking into account the GT definition: (2) τ-time delay, x-distance between radar and target, σ-the width of Gaussian windowed function. For the particular case of used IVS-192 MDR working in FMCW mode the Gabor spectrogram is presented in Fig.7.  [39] or Wi-Fi [21].
The IVS-948 presents better sensitivity, extended range, for monitoring the user during the rehabilitation. However, it is characterized by intensive power consumption, about three time more than IVS-142. Several tests were carried out when the target (user) is remotely monitored during the rehabilitation.

IV. THERMOGRAPHY
Thermography is a non-radiating and contact-free technology to monitor skin temperature that made this technology efficient in the case of muscular activity assessment during the rehabilitation period. One of the medical infrared thermography application reported in the literature is the monitoring of the athletes that are exposed to physical stress during physical training in order to avoid injuries [40]. Additionally, the use of thermography for rehabilitation outcomes assessment through the remote temperature measurement of the injuries' region are also reported in [41].
The novelty of the present approach is related to the usage of thermography for unobtrusive evaluation of muscle activity during serious game carried out in virtual and real environment or during physical rehabilitation using the walking aids. The temperature of the skin is measured before and after physical rehabilitation task using a thermographic camera FLIR E60. The main thermographic camera characteristics are: 320x240 thermographic images for a frame rate of 60Hz, thermographic sensitivity greater than 0.05ºC and the accuracy is 2% of reading for an extended measurement range between -20°C and 650°C.
The temperature measurement is based on thermographic image processing using FLIR Tools+ thermography image analysis toolkit [42]. Thus, a selection of specific regions of the recorded skin temperature images is performed in order to extract the minimum, maximum and average temperatures for each region. (Fig.8). Using the presented remote sensing technologies, a set of tests were carried out during the physical rehabilitation process. A set of serious game training sessions were considered for hand rehabilitation purpose where the interaction between user and VR scenario of the serious game is carried out using natural user interfaces expressed by Leap Motion and Kinect.
Taking into account that the Leap Motion mainly performs the acquisition of the finger position additional information can be extracted using the MDR that can evaluate the cardiorespiratory activity ( Fig.9.a, Fig. 9.b) for the user static condition (before and after the game) or can be used to extract the body motion and posture during the gaming times ( Fig.9.c, Fig.9.d). The muscles activity is carried out in unobtrusive way using thermography, thus the temperature measurement on the hand level was performed before and after the game session with Leap Motion.
In the case of Kinect serious game the MDR sensor is not required for body motion detection or posture measurement taking into account the capability of Kinect on posture detection but is can be useful for remote measurement of cardio-respiratory activity for static condition as was considered for Leap motion serious game. The MDR output signal evolution before and after Kinect Serious game is presented in Fig.10.
In order to characterize the upper limb patient rehabilitation outcome different metrics can be calculated starting from joints coordinates values. Thus the angle amplitude and joint velocities are calculated to figure out the evolution during rehabilitation process based on Kinect serious game. In Fig.  10 are presented the evolution of shoulder elbow right (SER) and shoulder elbow left (SEL) amplitudes in degree for 1 min training based on Kinect Therasoup serious game.  Several tests were carried out using the implemented serious game for Kinect. The number of points obtained during the training session was used to extract the hand motion rehabilitation progress considering the participation of four volunteers. The results obtained by the volunteers for three successive training session based on "Collect Color Cube" are presented in Table 1. Table 1. "Collect Color Cube" session score  ID\score  1 min session  2 min session  ID01  -1  3  ID02  0  20  ID03  -2  28  ID04  4  29 As can be observed the serious game players generally obtain a good score in the second session of 2 min.
The muscular activity was evaluated using thermography remote sensing technology. The temperature in specific region of the upper limb were measured before and after imposed physical rehabilitation task. From user to user under training were registered different variations of the temperature for the same imposed training tasks, however for individual patient under rehabilitation can be easily extracted information about the training intensity based on measured temperature. The temperature on the skin level in the region where the muscle were intensively used during the game period didn't underline a direct relation between the obtained number of points and the final temperature on the skin level, however for each player at the end of the session the temperatures were increased. As temperature values measured in remote way can be mentioned average temperature values starting of 26.1ºC before the training and at the end of the training are measured values up to 31.5ºC.
The hand training dexterity assessment based on Leap Motion natural interface was also performed. For one of the volunteers the result obtained for spiral test is presented in Fig. 12. velocity and coordination of movements can be obtained during these tasks.

VI. CONCLUSIONS
The presented remote sensing technologies associated to VR and AR serious game may have important role in the present and future developments in the field of physical rehabilitation. Thus new physical training procedures can be implemented, the progress of motor rehabilitation can be evaluated in objective way based on the information provided by the remote sensing systems. As the future work can be mentioned the integration of all presented remote sensing technologies in a physical rehabilitation platform characterized by multimodal interaction during serious game training sessions.