Video content analysis or video content analytics (VCA), also known as video analysis or video analytics (VA), is the capability of automatically analyzing

video Video is an Electronics, electronic medium for the recording, copying, playback, broadcasting, and display of moving picture, moving image, visual Media (communication), media. Video was first developed for mechanical television systems, whi ...

to detect and determine temporal and spatial events. This technical capability is used in a wide range of domains including entertainment,KINECT
, add-on peripheral for the

Xbox 360 The Xbox 360 is a home video game console developed by Microsoft. As the successor to the Xbox (console), original Xbox, it is the second console in the Xbox#Consoles, Xbox series. It was officially unveiled on MTV on May 12, 2005, with detail ...

console video retrieval and

video browsing Video browsing, also known as exploratory video search, is the interactive process of skimming through video content in order to satisfy some information need or to interactively check if the video content is relevant. While originally proposed to h ...

, health-care, retail, automotive, transport,

home automation Home automation or domotics is building automation for a home. A home automation system will monitor and/or control home attributes such as lighting, climate, entertainment systems, and appliances. It may also include home security such ...

, flame and smoke detection, safety, and security.VCA usage increase in British Security
, BSIA report The

algorithm In mathematics and computer science, an algorithm () is a finite sequence of Rigour#Mathematics, mathematically rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algo ...

s can be implemented as software on general-purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include

video tracking Video tracking is the process of locating a moving object (or multiple objects) over time using a camera. It has a variety of uses, some of which are: human-computer interaction, security and surveillance, video communication and compression, a ...

and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as

video summarization Automatic summarization is the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content. Artificial intelligence algorithms are commo ...

identification Identification or identify may refer to: *Identity document, any document used to verify a person's identity Arts, entertainment and media * ''Identify'' (album) by Got7, 2014 * "Identify" (song), by Natalie Imbruglia, 1999 * ''Identification ...

behavior Behavior (American English) or behaviour (British English) is the range of actions of Individual, individuals, organisms, systems or Artificial intelligence, artificial entities in some environment. These systems can include other systems or or ...

analysis, or other forms of

situation awareness Situational awareness or situation awareness, often abbreviated as SA is the understanding of an environment, its elements, and how it changes with respect to time or other factors. It is also defined as the perception of the elements in the envi ...

. VCA relies on good input video, so it is often combined with video enhancement technologies such as

video denoising Video denoising is the process of removing noise from a video signal. Video denoising methods can be divided into: * Spatial video denoising methods, where image noise reduction is applied to each frame individually. * Temporal video denoising metho ...

image stabilization Image stabilization (IS) is a family of techniques that reduce motion blur, blurring associated with the motion of a camera or other imaging device during exposure (photography), exposure. Generally, it compensates for panning (camera), pan an ...

unsharp masking Unsharp masking (USM) is an image sharpening technique, first implemented in darkroom photography, but now commonly used in digital image processing software. Its name derives from the fact that the technique uses a blurred, or "unsharp", negat ...

, and

super-resolution Super-resolution imaging (SR) is a class of techniques that improve the resolution of an imaging system. In optical SR the diffraction limit of systems is transcended, while in geometrical SR the resolution of digital imaging sensors is enhanced ...

Functionalities

Several articles provide an overview of the modules involved in the development of video analytic applications.Nik Gagvani
Introduction to Video AnalyticsCheng Peng
Video Analytics This is a list of known functionalities and a short description.

Commercial applications

VCA is a relatively new technology, with numerous companies releasing VCA-enhanced products in the mid-2000s. While there are many applications, the track record of different VCA solutions differ widely. Functionalities such as

motion detection In physics, motion is when an object changes its position with respect to a reference point in a given time. Motion is mathematically described in terms of displacement, distance, velocity, acceleration, speed, and frame of reference to an obser ...

people counting A people counter is an electronic device that is used to measure the number of people traversing a certain passage or entrance. Examples include simple manual clickers, smart-flooring technologies, infrared beams, thermal imaging systems, Wi-Fi trac ...

and gun detection are available as

commercial off-the-shelf Commercial-off-the-shelf or commercially available off-the-shelf (COTS) products are packaged or canned (ready-made) hardware or software, which are adapted aftermarket to the needs of the purchasing organization, rather than the commissioning of ...

products and believed to have a decent track-record (for example, even freeware such as dsprobotics Flowstone can handle movement and color analysis). In response to the

COVID-19 pandemic The COVID-19 pandemic (also known as the coronavirus pandemic and COVID pandemic), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), began with an disease outbreak, outbreak of COVID-19 in Wuhan, China, in December ...

, many software manufacturers have introduced new public health analytics like

face mask The face is the front of the head that features the eyes, nose and mouth, and through which animals express many of their emotions. The face is crucial for human identity, and damage such as scarring or developmental deformities may affect the ...

detection or

social distancing In public health, social distancing, also called physical distancing, (NB. Regula Venske is president of the PEN Centre Germany.) is a set of non-pharmaceutical interventions or measures intended to prevent the spread of a contagious dise ...

tracking. In many domains VCA is implemented on

CCTV Closed-circuit television (CCTV), also known as video surveillance, is the use of closed-circuit television cameras to transmit a signal to a specific place on a limited set of monitors. It differs from broadcast television in that the signa ...

systems, either distributed on the cameras (at-the-edge) or centralized on dedicated processing systems. Video Analytics and Smart CCTV are commercial terms for VCA in the security domain. In the UK the BSIA has developed an introduction guide for VCA in the security domain.British Industry VCA Guide
, 262 An Introduction to Video Content Analysis Industry Guide In addition to video analytics and to complement it, audio analytics can also be used. UK based startup that provides audio analytics into the CCTV industry Video management software manufacturers are constantly expanding the range of the video analytics modules available. With the new suspect tracking technology, it is then possible to track all of this subject's movements easily: where they came from, and when, where, and how they moved. Within a particular surveillance system, the indexing technology is able to locate people with similar features who were within the cameras’ viewpoints during or within a specific period of time. Usually, the system finds a lot of different people with similar features and presents them in the form of snapshots. The operator only needs to click on those images and subjects which need to be tracked. Within a minute or so, it's possible to track all the movements of a particular person, and even to create a step-by-step video of the movements.

Kinect Kinect is a discontinued line of motion sensing input devices produced by Microsoft and first released in 2010. The devices generally contain RGB color model, RGB cameras, and Thermographic camera, infrared projectors and detectors that map dep ...

is an add-on peripheral for the

gaming console that uses VCA for part of the user input. In retail industry, VCA is used to track shoppers inside the store. By this way, a heatmap of the store can be obtained, which is beneficial for store design and marketing optimisations. Other applications include dwell time when looking at a products and item removed/left detection. The quality of VCA in the commercial setting is difficult to determine. It depends on many variables such as

use case In both software and systems engineering, a use case is a structured description of a system’s behavior as it responds to requests from external actors, aiming to achieve a specific goal. It is used to define and validate functional requireme ...

implementation Implementation is the realization of an application, execution of a plan, idea, scientific modelling, model, design, specification, Standardization, standard, algorithm, policy, or the Management, administration or management of a process or Goal ...

system configuration A system configuration (SC) in systems engineering defines the computers, processes, and devices that compose the system and its boundary. More generally, the system configuration is the specific definition of the elements that define and/or presc ...

and

computing platform A computing platform, digital platform, or software platform is the infrastructure on which software is executed. While the individual components of a computing platform may be obfuscated under layers of abstraction, the ''summation of the requi ...

. Typical methods to get an objective idea of the quality in commercial settings include independent

benchmarking Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are Project management triangle, quality, time and cost. Benchmarking is ...

i-Lids
Benchmarking initiative by the

UK Home Office The Home Office (HO), also known (especially in official papers and when referred to in Parliament) as the Home Department, is the United Kingdom's interior ministry. It is responsible for public safety and policing, border security, immigr ...

and designated test locations. VCA has been used for

crowd management Crowd control is a public security practice in which large crowds are managed in order to prevent the outbreak of crowd crushes, affray, fights involving drunk and disorderly people or riots. Crowd crushes in particular can cause many hundre ...

purposes, notably at

The O2 Arena The O2 Arena, commonly known as The O2, is a multi-purpose indoor arena in the centre of The O2 entertainment district on the Greenwich Peninsula in southeast London, England. It opened in its present form in 2007. It has the third-highest seat ...

in London and

The London Eye The London Eye, originally the Millennium Wheel, is a cantilevered observation wheel on the South Bank of the River Thames in London. It is the world's tallest cantilevered observation wheel, and the most popular paid Tourist attractions in th ...

Law enforcement

Police and forensic scientists analyse

video when investigating criminal activity. Police use software, such as

Kinesense Kinesense is computer vision and video analytics company based in Dublin, Ireland. The company is one of largest suppliers of computer vision products to the UK police, who use the technology to search CCTV content in the course of criminal inves ...

, which performs video content analysis to search for key events in video and find suspects. Surveys have shown that up to 75% of cases involve CCTV. Police use video content analysis software to search long videos for important events.

Academic research

Video content analysis is a subset of

computer vision Computer vision tasks include methods for image sensor, acquiring, Image processing, processing, Image analysis, analyzing, and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical ...

and thereby of

artificial intelligence Artificial intelligence (AI) is the capability of computer, computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of re ...

. Two major academic benchmark initiatives are TRECVID,TRECVID
Academic benchmark initiative by

NIST The National Institute of Standards and Technology (NIST) is an agency of the United States Department of Commerce whose mission is to promote American innovation and industrial competitiveness. NIST's activities are organized into physical s ...

which uses a small portion of i-LIDS video footage, and the PETS Benchmark Data.PETS Benchmark Data
, Performance Evaluation of Tracking and Surveillance (PETS) by

University of Reading The University of Reading is a public research university in Reading, Berkshire, England. It was founded in 1892 as the University Extension College, Reading, an extension college of Christchurch College, Oxford, and became University College, ...

They focus on functionalities such as tracking, left luggage detection and virtual fencing. Benchmark video datasets such as th
UCF101
ref name="Center 2013"> enables action recognition researches incorporating temporal and spatial visual attention with

convolutional neural network A convolutional neural network (CNN) is a type of feedforward neural network that learns features via filter (or kernel) optimization. This type of deep learning network has been applied to process and make predictions from many different ty ...

and

long short-term memory Long short-term memory (LSTM) is a type of recurrent neural network (RNN) aimed at mitigating the vanishing gradient problem commonly encountered by traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, ...

. Video analysis software is also being paired with footage from body-worn and dashboard cameras in order to more easily redact footage for public disclosure and to identify events and people in videos. The EU is funding a FP7 project called P-REACTP-REACT Project Website
/ref> to integrate video content analytics on embedded systems with police and transport security databases.

Artificial Intelligence

Artificial intelligence for video surveillance utilizes

computer software Software consists of computer programs that instruct the Execution (computing), execution of a computer. Software also includes design documents and specifications. The history of software is closely tied to the development of digital comput ...

programs that analyze the audio and images from video surveillance cameras in order to recognize humans, vehicles, objects and events. Security contractors program is the software to define restricted areas within the camera's view (such as a fenced off area, a parking lot but not the sidewalk or public street outside the lot) and program for times of day (such as after the close of business) for the property being protected by the camera

surveillance Surveillance is the monitoring of behavior, many activities, or information for the purpose of information gathering, influencing, managing, or directing. This can include observation from a distance by means of electronic equipment, such as ...

. The

("A.I.") sends an alert if it detects a trespasser breaking the "rule" set that no person is allowed in that area during that time of day.

References