eprintid: 755
rev_number: 6
eprint_status: archive
userid: 17
dir: disk0/00/00/07/55
datestamp: 2019-05-11 14:02:04
lastmod: 2019-05-11 14:02:04
status_changed: 2019-05-11 14:02:04
type: report
metadata_visibility: show
creators_name: Ward, Jonathan
creators_name: Antonucci, Giancarlo
creators_name: Barter, Edmund
creators_name: Brooks-Tyreman, Frank
creators_name: Connaughton, Colm
creators_name: Coughlan, Michael
creators_name: Kuhne, Ronja
creators_name: Kaiser, Marcus
creators_name: Wang, Victor
corp_creators: Joel Woon
title: Evaluation of the Accuracy of a Computer-Vision Based Crowd Monitoring System
ispublished: pub
subjects: telecom
studygroups: ESGI138
companyname: CrowdVision
full_text_status: public
abstract: Computer vision systems can be used to measure pedestrian flow rates, occupancy levels and queue times. It is difficult to assess the accuracy of such methods because the ground truth can be difficult to establish. Human counting is equally prone to error, even when using video recordings with no time constraints and the support of sophisticated software.
In this report, we consider how errors may arise directly from the images recorded by the cameras, due to both occlusion of people and image distortion due to a fisheye lens. We also develop a statistical model of human counting errors and attempt to estimate human accuracy from data. Finally, we attempt to relate human and computer accuracy on the basis of simplifying statistical approximations.
problem_statement: CrowdVision sells a software solution to monitor crowd levels in enclosed spaces in real- time, their main installations being at large international airports. Their monitoring method relies on using computer vision from an array of fixed overhead cameras to estimate oc- cupancy levels, flow rates and queuing times within certain prescribed areas. One of the requirements of many of their clients is that they can demonstrate that their system is at least 95% accurate. However, it’s not clear how to measure the accuracy of their system since the ground truth count is not known. CrowdVision have found that manual counts can vary from person to person, even when using video software that can be paused and played multiple times. Thus both machine-generated counts and human counts are imper- fect. Moreover, the difference between two humans is often greater than the tolerance level required of the computer vision solution. It is therefore not possible to measure the system’s accuracy directly.
The desired outcome of the study group was to help CrowdVision improve the method by which they validate their solutions. In particular, we were asked to develop a statistically ro- bust protocol for verification of CrowdVision’s system using the minimum number of human counts. Crowd vision also posed the following mathematical questions:
• Can we understand the true sources of variability of human approaches to counting occupancy levels? Are there specific events or types of crowd movements that lead to specific kinds of increased variability?
• What are the main sources of error in the CrowdVision system? Is the nature of the overlap region between cameras and how these are dealt with the main source of error? How does the error in the crowd size vary with the number of cameras used? Is a boot- strapping approach to error estimation possible?
• Can we look at specific errors in rate and queuing times? What is the main source of error when the camera is not directly overhead to the end of the queue?
Our work during the study group consisted of two distinct parts. In the first part, the ge- ometric analysis described in Section 5, we considered how errors might arise due to the occlusion of people within the line of sight of a camera, and due to distortion effects re- sulting from the camera’s fish eye lens. In the second part, the data analysis described in Sections 6 and 7, we analysed anonymised count data provided by CrowdVision and devel- oped statistical models to relate the difference between both human and machine counts to estimate the repsective errors.
date: 2018
citation:   Ward, Jonathan and Antonucci, Giancarlo and Barter, Edmund and Brooks-Tyreman, Frank and Connaughton, Colm and Coughlan, Michael and Kuhne, Ronja and Kaiser, Marcus and Wang, Victor  (2018) Evaluation of the Accuracy of a Computer-Vision Based Crowd Monitoring System.  [Study Group Report]     
document_url: http://miis.maths.ox.ac.uk/miis/755/1/esgi-crowdvision-draft-v1.pdf