Abstract
nning gait assessment is essential for the development of technical optimization strate- gies as well as to inform injury prevention and rehabilitation. Currently, running gait assessment relies on (i) visual assessment, exhibiting subjectivity and limited reliability, or (ii) use of instrumented approaches, which often carry high costs and can be intrusive due to the attachment of equipment to the body. Here, the use of an IoT-enabled markerless computer vision smartphone application based upon Google's pose estimation model BlazePose was evaluated for running gait assessment for use in low-resource settings. That human pose estimation architecture was used to extract contact time, swing time, step time, knee exion angle, and foot strike location from a large cohort of runners. The gold-standard Vicon 3D motion capture system was used as a reference. The proposed approach performs robustly, demonstrating good (ICC(2,1) > 0.75) to excellent (ICC(2,1) > 0.90) agreement in all running gait outcomes. Additionally, temporal outcomes exhibit low mean error (0.010.014 s) in left foot outcomes. However, there are some discrepancies in right foot outcomes, due to occlu- sion. This study demonstrates that the proposed low-cost and markerless system provides accurate running gait assessment outcomes. The approach may help routine running gait assessment in low-resource environments. Keywords: gait analysis; computer vision; deep learning; signal
temporal outcomes exhibit low mean error (0.010.014 s) in left foot outcomes. However, there are some discrepancies in right foot outcomes, due to occlu- sion. This study demonstrates that the proposed low-cost and markerless system provides accurate running gait assessment outcomes. The approach may help routine running gait assessment in low-resource environments. Keywords: gait analysis; computer vision; deep learning; signal analysis; pose estimation; BlazePose; smartphone application 1. Introduction Running for sport and exercise has continually grown in popularity due to its obvious health bene ts and low (cost, skill, resource) barrier to entry [1]. As participation in running has increased, so too has the assessment and analysis of running gait to, e.g., avoid injury or improve ef ciency. Running gait assessment generally encapsulates the study of lower- extremity kinematics [2]. Particularly, running gait assessment is paramount in providing injury prevention and rehabilitation mechanisms through quantifying the factors pertaining to the unique patterns of limb movement and co-ordination [3]. For example, runners exhibiting a rear-foot strike have been found to be at almost twice the risk of a repetitive strain injury in comparison to those with a fore or mid-foot strike [4]. As such, identifying and adapting small changes such as strike location to the running stride can minimize injury risk. Additionally, identifying shortcomings and adapting the running stride can lead to performance optimizations and overall better technique and speed. For example, adopting a forefoot strike location can minimize ground contact time, contributing to an increased race speed [5], and improve metabolic economy [6]. Traditionally, running gait assessment has relied upon manual, visual/video-based observation of treadmill running by a trained individual (e.g., bio-mechanist, sports ther- apist). However, there is some dispute as to the reliability of such approaches between assessors, often being reliant upon the observed plane and/or assessor experience [7]. Sensors2023,23, 696.
Sensors2023,23, 696 2 of 16 Consequently, running gait assessment has moved towards instrumented approaches such as wearable technology including inertial measurement units (IMU), force/pressure plate analysis [810], as well as three-dimensional (3D) motion tracking [11,12] in an effort to provide reliable, reproducible outcomes. Despite research-grade wearable technology's utility in providing a wide range of gait outcomes, they are currently limited in use due to the cost [13], tethering of peripheral technologies, and reliance upon bespoke environments with expert assistance [9]. For example, wearable sensors or 3D-motion re ective markers often rely upon precise anatomical placement for optimal use [14,15]. Additionally, such technologies are naturally intrusive, often requiring adhesion to the skin or clothes by, e.g., medical tape [16], which may cause discomfort and thus, affect a natural running cycle; limiting the usability of such approaches for habitual use. Consequently, investigation into the use of markerless assessment systems for running gait assessment is warranted. Markerless systems typically encapsulate human pose estimation methods from 2D or 3D video streams. For example, OpenPose [17] utilizes deep learning-based part af nity elds to identify anatomical locations from a 2D video stream, providing a digital skeleton of an individual performing a range of tasks. Alternatively, Microsoft's Kinect [18] utilizes depth sensors and infrared cameras to provide a 3D mapping of a human skeleton in near real-time. Although 3D approaches such as Kinect have demonstrated validity in certain spatiotemporal gait outcomes [19], their innate high cost (e.g., speci c hardware rather than a generic camera) in comparison with 2D-based deep learning approaches limit their use much like wearable and marker-based tracking technologies. Accordingly, a 2D-based approach is generally preferred (and is the focus of this study). To date, 2D markerless pose estimation has primarily seen research interest within normal walking/gait assessment. For example, markerless pose estimation has been in- vestigated for use within clinical-based studies [2022], where computer vision-based gait assessment could be used as a primary biomarker to assess disease onset and well-being. In the referenced studies, both spatial [2022] and temporal [21] outcomes are evaluated with generally positive results. In contrast, there are signi
interest within normal walking/gait assessment. For example, markerless pose estimation has been in- vestigated for use within clinical-based studies [2022], where computer vision-based gait assessment could be used as a primary biomarker to assess disease onset and well-being. In the referenced studies, both spatial [2022] and temporal [21] outcomes are evaluated with generally positive results. In contrast, there are signi cantly fewer developments for running gait with 2D pose estimation, although the use of OpenPose has demonstrated an ability to examine cadence [23]. Yet, within running gait, there are a signi cantly higher number of relevant outcomes to inform, e.g., performance optimization such as contact time, swing time, and knee exion angle [24,25]. Accordingly, this warrants further explo- ration of the validity of markerless pose estimation within running gait. Furthermore, any contemporary approach to analyzing running gait should ensure exibility to assess in any environment, beyond bespoke facilities to evoke natural running patterns [26]. Accord- ingly, reliance upon the OpenPose infrastructure, despite its reliability, requires signi cant computational cost (e.g., multi-GPU, high-powered computer) and so limits use within habitual or low-resource settings. In contrast, routine technologies (e.g., smartphones) may be more suitable. As such, low-powered 2D approaches must be investigated to enable running gait assessment beyond complex, research-grade environments. BlazePose is a low-powered markerless pose estimation technique and has recently demonstrated the ability to run on inexpensive hardware (Pixel 2 smartphone) [27]. The approach has been shown to provide a tradeoff relationship with OpenPose, sacri cing some anatomical accuracy for a signi cant reduction in computational cost in clinical environments [28]. Consequently, the approach could augment assessments within low- resource settings through deployment on low-powered hardware such as a smartphone. Considering approx. 90% of adults own a smartphone [29], and the technology is generally ubiquitous in everyday life. Accordingly, smartphones are providing a mecha- nism for remote healthcare, data capture, and transmission within the Internet of Things (IoT) [30] due to their relatively low computational power, connectivity and storage capa- bilities. Equally, a smartphone can capture data, while of oading complex computations to an external device (e.g., server). Consequently, smartphones
the technology is generally ubiquitous in everyday life. Accordingly, smartphones are providing a mecha- nism for remote healthcare, data capture, and transmission within the Internet of Things (IoT) [30] due to their relatively low computational power, connectivity and storage capa- bilities. Equally, a smartphone can capture data, while of oading complex computations to an external device (e.g., server). Consequently, smartphones could be used to augment complex assessments that were previously reliant on expert in-person visits. Applied to running gait assessment, a smartphone was conceptualized as the data capture mechanism
Sensors2023,23, 696 3 of 16 (via video) and then transmitted to a local edge device to enable a more accessible running gait assessment. As such, here, a single-camera approach to running gait assessment uti- lizing the low-powered BlazePose 2D pose estimation framework is proposed. The study aims to assess the validity of the low-powered approach, running on a custom smartphone application and low-cost server in comparison to a reference-standard 3D motion tracking system. Through validating the low-resource approach, the work aims to contribute to moving the eld of running gait assessment out of bespoke facilities (e.g., lab) and into low-resource (e.g., everyday/habitual) environments by providing a simpli ed mechanism for running gait assessment (smartphone camera) in comparison to existing gold-standard approaches, that could conceivably be used, e.g., in the home, or at the gym. 2. Materials and Methods 2.1. Participants Thirty-one healthy, experienced runners (34.5 9.7 years; 1.75 0.30 m;76.2 4.1 kg ; 20 male:11 female) were recruited from running clubs throughout the Northeast of England. Participants were screened for previous running-related injuries (RRI) and their ability to perform unsupervised, short treadmill-based running bouts. No participants reported any gait-affecting injuries or conditions that would adversely affect their ability to participate in the study and all had previous treadmill running experience. Ethical approval for the study was granted by Northumbria University's Research Ethics Committee (reference: 21603). Prior to testing, participants were given informed consent and provided verbal and written consent following a short brie ng. Upon con- senting, participants were provided with a standardized, neutral cushioning running shoe (Saucony Guide Runner) to wear during testing in order to remove bias from gait-affecting cushioning within, e.g., support cushioning running shoes [31]. 2.2. Video Capture Participants were video recorded from the side and rear angles during treadmill running sessions utilizing two iPhone 13 smartphones, capturing at 240 frames per second (FPS) to provide slow-motion video streams for the application of 2D pose estimation. The smartphones were placed in separate static mounts approx. six feet from the left side and rear of the treadmill, standardizing the video capture sessions. Before running, participants were lmed
rear angles during treadmill running sessions utilizing two iPhone 13 smartphones, capturing at 240 frames per second (FPS) to provide slow-motion video streams for the application of 2D pose estimation. The smartphones were placed in separate static mounts approx. six feet from the left side and rear of the treadmill, standardizing the video capture sessions. Before running, participants were lmed performing a short static test, where they stood up straight with their hands to the side to establish a baseline reading. Once calibrated, participants performed up to ve 1 min runs on a treadmill between 8 km/h and 14 km/h, selected based upon a pace comparable to their most recent outdoor 5 km pace. 2.3. Reference System and Data Labelling To provide a ground truth to benchmark the system performance against, a 14-camera 3D motion tracking system (Vicon Vertex, UK,, accessed on 28 November 2022) was used, providing a high-resolution 3D skeletal mapping of participants during treadmill running. Participants were tted with 16 neo-re ective markers, located at the calcaneal tuberosity (heel), lateral malleoli (ankle), base of the second metatarsal (front- foot/toe), lateral mid-shank, lateral knee joint line, mid-lateral thigh, anterior superior iliac spine, and posterior superior iliac spine. The Vicon 3D motion tracking system was con gured to poll at 200 Hertz (Hz) to provide a suitable rate of detail for intricate gait assessment. To ensure robust outcomes were obtained from the Vicon system, the built-in gait analysis suites were utilized, which have been used to provide a gold-standard set of gait outcomes for comparison with other sensing technologies such as infrared cameras [32] and wearable technology [33]. The following gait outcomes were obtained from the Vicon system. Initial contact (IC)the point at which the foot rst contacts the ground. Final contact (FC)the point at which the foot rst leaves the ground.
Sensors2023,23, 696 4 of 16 Contact time (CT)the total time elapsed between IC and FC (i.e., time foot spent in contact with the ground. Swing time (ST)the time elapsed between an FC event and a proceeding IC (i.e., time foot spent off the ground). Step time (StT)the time elapsed between two IC events. Cadencethe number of steps taken per minute of running. Knee exion anglethe angle between lateral mid-shank, lateral knee joint line, and mid-lateral thigh throughout a gait cycle. Foot strike locationthe angle of the foot during contact with the ground during IC. 2.4. Proposed Low-Cost Approach Typically, OpenPose [17], has been used for 2D computer vision-based running gait assessment but requires a complex GPU con guration due to reliance on convolution pose machines [34] and part af nity elds [35], which may create a barrier of entry to typical users through high associated computational costs. Furthermore, through utilizing part af nity elds, OpenPose performs multi-person pose detection, which creates additional complexities that are unnecessary within running gait assessment (for a single individual). Alternatively, Blazepose provides 33 anatomical key points for single-person videos, which include hips, knees, ankles, heels, and toes. That provides a suitable level of anatomical detail for understanding running gait biomechanics [36], while retaining a substantially lower computational cost by naively relying on identi cation of the head and its relative position to the body [27]. By utilizing the naïve approach, the BlazePose architecture could in turn augment gait assessment on low-cost hardware such as a smartphone or within an edge computing context, especially when considered against, e.g., OpenPose. 2.4.1. Proposed Infrastructure To maximize the utility of the proposed 2D approach, a smartphone application was developed, interfacing with a Raspberry Pi server where pose estimation and analysis take place. By utilizing a smartphone in combination with IoT-based edge processing, the system could in turn be adapted for full-scale development/release, lowering the barriers of entry to running gait assessment. An overview of the infrastructure can be found in Figure, detailing the ow from a custom smartphone application to the Raspberry Pi-based server, where a cloud processing unit
analysis take place. By utilizing a smartphone in combination with IoT-based edge processing, the system could in turn be adapted for full-scale development/release, lowering the barriers of entry to running gait assessment. An overview of the infrastructure can be found in Figure, detailing the ow from a custom smartphone application to the Raspberry Pi-based server, where a cloud processing unit extracts gait outcomes for display on the smartphone application.Sensors 2023, 23, 696 5 of 18 Figure 1. The infrastructure of the smartphone application and its cloud interface. Here, (A) the smartphone application captures and uploads side-view videos, transmitting to (B) a cloud pro- cessing unit running on a Raspberry Pi 4 where a Python Django implementation of Blazepose esti- mates and (C) stores the location of anatomical locations/keypoints, wherein signals are analyzed for gait outcomes. (D) Outcomes are then returned to the user via JSON for interpretation in the smartphone application. 2.4.2. Smartphone Application and Cloud Infrastructure Here, a cross-platform (iOS, Android) smartphone application was developed with the Flutter 2.0 software development kit (SDK) wherein users can capture and upload 1 min (left) side-view videos of treadmill running. Selected videos are compressed using H.264 compression to optimize data transmission to a low-powered cloud infrastructure, wherein analysis takes place. Despite Blazepose’s ability to perform on-device pose estimation, the approach still requires reasonably modern hardware (e.g., iOS version 12+). As such, a low-powered, low-cost cloud infrastructure was developed utilizing a Raspberry Pi 4 (GBP ≈ £35) and Python’s Django library to maximize the compatibility with a larger number of smartphone devices (Figure 1). The cloud infrastructure first receives videos from the smartphone application and temporarily stores them. Then, the video is run through a BlazePose instance within a Django server, wherein the X and Y locations of relevant an- atomical key points of each frame are stored in a .CSV file. The generated .CSV file is then analyzed by the running gait feature extraction layer; wherein extracted outcomes are ren- dered as a JSON object for transmission to the smartphone application for interpretation, i.e., running assessment. Following a successful analysis, the video
X and Y locations of relevant an- atomical key points of each frame are stored in a .CSV file. The generated .CSV file is then analyzed by the running gait feature extraction layer; wherein extracted outcomes are ren- dered as a JSON object for transmission to the smartphone application for interpretation, i.e., running assessment. Following a successful analysis, the video is deleted from the cloud storage to maintain user privacy. 2.5. Feature Extraction The feature extraction layer of the proposed system was performed using libraries that are commonplace within Python’s data science and computer vision capabilities (Fig- ure 1). Particularly, OpenCV [37] handles the video stream playback and metadata, where SciPy, Pandas, and NumPy [38,39] were used in combination for data manipulation, sig- nal analysis, statistics, and other mathematical operations detailed throughout the paper. The following section details how running gait features are extracted from the X and Y locations of anatomical keypoints utilizing signal analysis. 2.5.1. Data Preprocessing Occasionally, keypoints identified by BlazePose can behave erratically and may be subject to noise that is not indicative of the real position of the anatomical location. Con- sequently, applying a preprocessing layer to the signals is paramount for accurate Figure 1. The infrastructure of the smartphone application and its cloud interface. Here, (A) the smartphone application captures and uploads side-view videos, transmitting to (B) a cloud processing unit running on a Raspberry Pi 4 where a Python Django implementation of Blazepose estimates and (C) stores the location of anatomical locations/keypoints, wherein signals are analyzed for gait outcomes. (D) Outcomes are then returned to the user via JSON for interpretation in the smartphone application.
Sensors2023,23, 696 5 of 16 2.4.2. Smartphone Application and Cloud Infrastructure Here, a cross-platform (iOS, Android) smartphone application was developed with the Flutter 2.0 software development kit (SDK) wherein users can capture and upload 1 min (left) side-view videos of treadmill running. Selected videos are compressed using H.264 compression to optimize data transmission to a low-powered cloud infrastructure, wherein analysis takes place. Despite Blazepose's ability to perform on-device pose estimation, the approach still requires reasonably modern hardware (e.g., iOS version 12+). As such, a low-powered, low-cost cloud infrastructure was developed utilizing a Raspberry Pi 4 (GBP £35) and Python's Django library to maximize the compatibility with a larger number of smartphone devices (Figure). The cloud infrastructure rst receives videos from the smartphone application and temporarily stores them. Then, the video is run through a BlazePose instance within a Django server, wherein the X and Y locations of relevant anatomical key points of each frame are stored in a .CSV le. The generated .CSV le is then analyzed by the running gait feature extraction layer; wherein extracted outcomes are rendered as a JSON object for transmission to the smartphone application for interpretation, i.e., running assessment. Following a successful analysis, the video is deleted from the cloud storage to maintain user privacy. 2.5. Feature Extraction The feature extraction layer of the proposed system was performed using libraries that are commonplace within Python's data science and computer vision capabilities (Figure). Particularly, OpenCV [ 37] handles the video stream playback and metadata, where SciPy, Pandas, and NumPy [38,39] were used in combination for data manipulation, signal analysis, statistics, and other mathematical operations detailed throughout the paper. The following section details how running gait features are extracted from the X and Y locations of anatomical keypoints utilizing signal analysis. 2.5.1. Data Preprocessing Occasionally, keypoints identi ed by BlazePose can behave erratically and may be subject to noise that is not indicative of the real position of the anatomical location. Conse- quently, applying a preprocessing layer to the signals is paramount for accurate analysis. Here, a 5-step moving average is applied to each signal stream, optimally
keypoints utilizing signal analysis. 2.5.1. Data Preprocessing Occasionally, keypoints identi ed by BlazePose can behave erratically and may be subject to noise that is not indicative of the real position of the anatomical location. Conse- quently, applying a preprocessing layer to the signals is paramount for accurate analysis. Here, a 5-step moving average is applied to each signal stream, optimally selected by manual observation of the signal, ensuring noise is reduced while minimizing loss of signal delity. 2.5.2. Gait Mechanics: Identifying Key Features Initial contact and nal contact are key biomarkers within gait as they inform a vast range of temporal and angle gait features. As such, quantifying IC and FC before other features will augment the rest of the gait feature extractions. Within a running stride, IC and FC typically proceed with a maximum extension of the leg [40], wherein the angle between the hip, knee, and ankle is at its maximum value (tending towards 180 ), referred to here as the leg extension angle. To calculate the leg extension angle, locations are converted to vectors for vector scalar product analysis such that: ! KH=K H and ! KA=K A (1) whereK= the location of the knee,H= the location of the hip, andA= the location of the ankle extracted by the pose estimation layer. ! KH refers to the vector between knee and hip, whereas ! KA refers to the vector between knee and ankle, withKbeing the intersecting point
Description
This study evaluates a smartphone application for running gait assessment using IoT-enabled markerless technology.