Translational Research: What I learned doing (seemingly) mundane task of video annotation

In the recent past I have been doing some work related to automatic video annotation. Videos that you and I take can be annotated with data about the contents of the video. The contents of the video can mean: objects, their types, their shape, background scene (moving or static), number of objects, static and in-motion objects, color of objects etc. One would like to keep a track of objects as the video progresses. Tracking helps in knowing when an object appeared in the scene and when it disappeared. All of the prior work on automatic video annotation is not really completely automatic [1], [2] etc.. They are semi-automatic at best and manual input and control is still required when annotating using these methods.

While doing this work, I developed a better understanding of some of the so called “automatic object tracking for surveillance” solutions out there in the market.  None of these solutions can ensure a complete hands-off scenario for humans. Humans still need to be involved and there are reasons for that.  At the same time, it is also possible to do everything in cloud (including human interaction) and claim it as “hands off for a user”. In this case, it is simply that the client is paying someone else to provide the service. It is not a stand-alone autopilot kind of system installed in a user’s premises. Real automatic video annotation is extremely hard, especially when the scene can change without any guarantees. If we add “video analytics” i.e. ability to analyse the video automatically to detect a certain set of activities, it again becomes very difficult to propose a general solution. So, assumptions are again made and these can be based on user requirements or can be domain specific (say tennis video analytics at Wimbledon). Here is a system which may be of interest to you: IBM’s Digital Video Surveillance Service and a few others described in the paper titled “Automated visual surveillance in realistic scenarios“.

Most of the research work makes certain assumptions either about the scenes or about the methods they use. These assumptions simply fail in real world scenarios. These methods may work under a “restricted real world view” made using a set of assumptions, but when assumptions fail, these methods become limited in applicability.

I believe this is a critical issue that many researchers who want to translate their work into usable products have to understand. This is where both strong theoretical and practical foundations in a discipline are needed: theory gives the methods and the tools, engineering tells you what can/cannot be done and the two can interact back and forth.

Velocity, Displacement & Acceleration: Science vs. Engineering

One often encounters the question: What is the difference between science and engineering? An oft quoted answer is that engineering involves, roughly speaking, an application of science or scientific results borne out of investigation into the nature of matter and its interaction with its surroundings. Science is about acquiring more knowledge and understanding about existing phenomena whereas engineering involves solving problems by applying that knowledge. Therefore, many also hold the view that it is applied science. Well, I won’t get into the debate of engineering vs. science or put before you an essay on this topic in this post. I would just like to highlight an example of where engineering takes over from science. Every student studies the concepts of velocity, acceleration and displacement in elementary Physics classes. These concepts are very simple: velocity is the derivative of displacement with respect to time while acceleration is the derivative of velocity with respect to time. Therefore to get displacement from velocity , one needs to integrate the former with respect to time over a given time period. Similarly, velocity at a certain point in time is the result of integration of acceleration over a given time interval. Now, if one is asked to apply these principles to calculate velocity and displacement using the acceleration data obtained from a transducer mounted on an engine, how would one do it? In this case, the engine vibrates and there is no physical noticeable movement of engine body from one place to another in the traditional sense (like a ball traveling from place A to place B in a field). This is where engineering comes in. An engine is a complex system and its vibrations need not be linear or constant in time. There can be vibrations with low frequencies as well as high frequencies and there can be periods of no vibration at all. In these cases, calculation of displacement or velocity is not straight forward and requires greater insight into the mechanism of vibration as well as the nature of acceleration signal. I would recommend reading up 1, 2 and 3 to get an idea of how interesting and insightful it can become! These are links to articles by Prosig  which works in the area of noise and vibration analysis. Understanding these mechanisms is important for any embedded designer who writes code to measure such parameters using microcontrollers etc.