Stereo Vision-Based Human Tracking for Robotic Follower

Stereo Vision-Based Human Tracking for Robotic Follower

1. Introduction

   • Topico: Human detection and tracking has been extremely active research area in the computer vision community over the past decade.
     • Importancia: The importance of this area arises from its numerous applications such as video surveillance, smart vehicle and virtual
     • Ejempo: One of the broadest application areas is robot vision [2]. Vision‐ based Ejehuman tracking is crucial for human‐robot interaction in numerous human‐centred robot applications
       • Person following robots developed until now use various types (Hasta ahora utilizan varios tipos)of cameras for detecting a target person and some of them use (Algunos de ellos utilizan) other sensors in combination with vision. For example, in [3] a stereo vision system and additionally a Laser Range Finder (LRF) are mounted on the robot body to enhance the performance of the person following robot’s behaviour. Furthermore, some authors developed methods including human‐body wearable sensors [4], which are however, inadequate in some applications. In order to avoid (con el fin de evitar) the complex data fusion algorithms necessary in the case where different sensor types are used and in order to avoid human contact sensors, a number of authors have developed methods based on vision as the only sensor.
         • Although (A pesar de que)people detection and tracking with a single camera is a well‐explored topic, it has been shown that detecting humans from a single image while maintaining a low false detection rate is a very difficult problem [5]. The use of stereo vision provides a higher grade of information that brings several advantages when developing human‐robot applications. The information regarding disparities becomes more invariable (Sobre las disparidades se hace más invariable) to illumination changes than the images provided by a single camera, which is a very advantageous factor for environment estimation. Furthermore, the possibility of knowing the distance to the person could be of great assistance in tracking. The robot calculates the distance between the target person and itself using stereo vision and follows him/her with the appropriate speed to maintain the distance. In [6] a robust human detecting and tracking
     • The remainder of the paper is organised as follows. Section 2 presents the layout of the vision‐based robotic human follower system. Section 3 presents the first part of the vision module of robotic follower: stereo vision‐based human detection. The second part of the vision module of the robotic follower; human tracking based on a modified Kalman filter, is presented in Section 4. The performance evaluation of the presented stereo vision‐based human tracking within the working scenario of a mobile robot intended to follow a human co‐worker in indoor applications, as well as in outdoor applications, is presented in Section 5.

2. Body

   • Vision‐based robotic human follower – system layout
     • The presented vision‐based human tracking system is supposed to provide sensor input for vision‐based control of a mobile robot, which works in a team helping a human co‐worker with transportation of different objects in indoor applications, as well as in outdoor applications, such as investigation of a hazardous environment. For example, in the robot working scenario of investigation of contaminated/hazardous environments, the robot works as a transportation robot, helping the human to carry containers with collected samples from the environment. The vision system for human tracking within such (dentrode)robot working scenarios has to be able to detect the human, calculate the distance to the human and track the human, keeping a constant distance between them. After sensing the reduction in distance, indicating the human’s intention to approach the robot, the robot has to stop and allow the human to place the contain
     • The presented robotic system has a stereo camera system on board, as well as a low‐power industrial PC (Intel i7‐ 620M @ 2.66 GHz, 4GB of RAM, 64 GB SSD, ~65 Watt). As the platform is intended to run on batteries so as to move independently of external power supplies, low power consumption of the on‐board components is critical. Therefore, the on‐board PC functionality is limited to capturing images from the stereo camera and sending them compressed over the wireless link to the off‐board vision module, as well as sending direct commands obtained from the robot control to the wheel controllers.
     • In order to assure the control commands are sent at regular time intervals, the vision module and the robot control module run on different PCs, as illustrated in Figure 1, with differently coloured blocks. The computationally expensive vision algorithms run on a dedicated high‐end PC (Intel Xeon E5520 @ 2.26 GHz, 6GB of RAM) with the goal of tracking the human co‐ worker in front of the robot and determining its 3D position with respect to the coordinate system of the left stereo camera, as depicted in Figure 1. The obtained 3D position is then sent to the robot control module located on a different desktop, PC (Intel E4700 @ 2.6 GHz, 2GB of RAM), which computes the required velocities for each wheel such that the robot keeps following the human. These velocities are then sent at regular time intervals over the wireless link to the on‐board PC, which sends di
     • The communication between the three computers is carried out via the ROS (Robot Operating System), which is a widely used communication framework that, among other features, (Entre otras características,)allows easy configuring of multiple computers for cooperation in order to improve the time effectiveness of the system. This time effectiveness is achieved by splitting computationally expensive tasks into modules, which run on different computers. The focus in this paper is on the robot’s vision module. It consists of two sub‐modules: stereo vision‐based human detection and human tracking based on a modified Kalman filter.
   • Stereo vision‐based human detection
     • Stereo image information is used to aid the definition of regions of different objects, including humans, in camera images. The acquired image pair is used to compute a disparity map using a block‐matching algorithm [8]. In principle, the disparity map is found by computing the stereo correspondences between the image points from the left and right stereo images. For a given 3D point P(X, Y, Z), the corresponding points in the left and right image respectively are p L (u L ,v L )and p R (u R ,v R ), where u and v are coordinates of an image point in the image coordinate system with the origin in the camera’s optical centre. The difference in u coordinates of corresponding points in stereo images is known as disparity d.
       • The disparity of an image point is inversely proportional to the distance of the original 3D point to the camera coordinate system known as depth. The resulting disparity map represents a 2D image in which values of pixels are equal to the disparity (1). The pixel coordinates in a disparity map correspond to the pixel coordinates in the left stereo image as the left stereo image is usually considered as the reference image when building the disparity map.
     • The resulting disparity map in the proposed system is segmented using a connected pixel labelling‐based method. The main idea behind this segmentation method is to group the pixels with the same or very close pixel values as it is assumed that they belong to the same object. Namely, neighbouring pixels in the disparity map belonging to the surface of an object have close disparity values, while on the edges of the object the difference in disparity values between the pixels of the object and of the background is large. These transitions in disparity values are used for the segmentation. The details of the disparity map segmentation method used are given in [9]. The segmentation result in the case of the human tracking robot scenario considered here is shown in Figure 3(c). Differently coloured regions in the image in Figure 3(c) represent different objects, which are at differe
   • Performance evaluation
     • The performance of the presented stereo vision system for human tracking was tested within the working scenario of a mobile robot intended to follow a human co‐ worker in indoor applications, as well as in outdoor applications.
       • Experiments were conducted where a human walking in front of the mobile robot was imaged by a Point Grey Bumblebee XB3 [14] stereo camera mounted on the robot. The image pairs were grabbed at full resolution at 12fps (frames per second). Each pair of stereo frames was processed in order to extract information for stereo vision‐based reconstruction of human walking with respect to a camera coordinate system. Some examples of processed images with superimposed extracted bounding boxes of humans are shown in Figure 5. The result of human detection in five frames from the video captured indoors is shown in Figure 5 (a)‐(e). Figure 5 (f)‐(j) shows the result of human detection in five frames from the video captured outdoors.
       • In order to evaluate the performance of the system (Con el fin de evaluar el rendimiento del sistema)with respect to the accuracy of the reconstruction of the 3D coordinates of the person and therefore of the reconstruction of robot’s distance to the person, the experimental results were compared with the ground truth obtained in two ways. In the first experiment the ground truth distance to the person was obtained by a Bosch PLR 50 digital laser rangefinder [15]. In the second experiment a reference path to be followed by the person was drawn on the floor. As the focus in this paper is the presentation of the vision module of the robotic follower, in the performed experiments the robot only observes the person without following him/her so that the errors possibly occurring in vision can be decoupled from possible errors introduced by the robot control.
         • was compared with the distance obtained by a Bosch PLR 50 digital laser rangefinder. According to the datasheet, the accuracy of this device is ±2mm regardless of distance. The system was tested on 1055 frames and the average error in distance was 2.38% at a standard deviation of 2.12 %.

3. Conclusions

   • In this paper the vision module of a person‐following robot is presented. It consists of two sub‐modules: stereo vision‐based human detection and human tracking based on a modified Kalman filter. The robustness of human detection is provided by combining disparity (depth) image‐based features and reconstructed 3D human
     • features. The Kalman filter‐based prediction enables performing the human detection over the image region of interest rather than over the whole image, which makes the proposed human tracking method cost effective. The cost effective vision method is supported by distributed computing to support real‐time performance of robot control. Experimental results on detection of humans in videos captured in both environments, indoor and outdoor, are given. The performance evaluation was carried out with respect to the accuracy of human tracking, as well as with respect to cost effectiveness of the presented vision module.