Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am stuck trying to learn how to use video processing as explained in the linked papers in the area of human behavior detection or traffic surveillance (any kind of monitoring activity). In particular, I do not know how to formulate the problem using any available models.

I have found two papers (Haag & Nagel (2000; Paper 1) and Arens, Gerber, & Nagel (2008; Paper 2)) which illustrate, with an example, using natural language for traffic monitoring by using image sequences of videos. My question are

  1. Under what assumptions are the actions generated?What is the advantage of nlp here or in any application apart from the fact that it gives a hierarchical representation of information/knowledge?
  2. How are the actions and graphs (labelled as Situation Graph Trees by the author) generated (under what assumptions) in Fig2 of Paper 1 and Fig1 of Paper 2?
  3. Has the Table5 in paper1: Incremental recognition of traffic situations from video image sequences and subsequent tables been generated using some programming tools?


Haag, M., & Nagel, H.-H. (2000) "Incremental recognition of traffic situations from video image sequences". Image and Vision Computing 18(2): 137-153.

Arens, M., Gerber, R., & Nagel H.-H. (2008) "Conceptual representations between video signals and natural language descriptions". Image and Vision Computing 26: 53–66.

share|improve this question
I can't understand what you are asking. Also, if you don't what "stuff" you are doing, how can you ask for open-source tools for doing it? Figure out what you want to do and how to phrase it precisely and accurately in the form of one question, then based on what you learn in your initial search ask a specific well-focused questions. At this point I have to vote to close as "not a real question". – Artem Kaznatcheev Jun 5 '12 at 19:22
Your question is a confusing because of some unclean grammar as well as lack of focus. I think you should first try to separate this into two questions. One about the visual classification (maybe you want to look into sign language translation, visual recognition of gestures etc.). After you find something more interesting you want focus on, it will be easier to ask about open source tools for it. – Vielle Jun 5 '12 at 20:27
For example work on gesture recognition – Vielle Jun 5 '12 at 20:34
I edited the question to improve grammar and formatting so that OP could then edit it to make it more precise. There are actually two questions, in both the scope of desired robot cognition being overly broad. Some more limited scope of robot cognition would be IMHO a more fruitful point of departure for such a difficult matter. – nrz Jun 6 '12 at 19:13
OP, I think you should familiarize yourself with the basics of NLP. For some introductory resources, see this question and answers on ling.SE – Artem Kaznatcheev Jun 6 '12 at 21:14

I think you misunderstood, in a lot of ways, what Haag & Nagel (2000; what you call Paper 1) did and how Arens, Gerber & Nagel (2008; Paper 2) extended it. Fig. 1 of AGN08 is a good summary of HN00. What HN00 did was build a system that could watch a video of an intersection, detect cars, and translate the car behavior into a conceptual framework. As inspiration for their system, they used their idea of how humans represent the task:

five levels of representation seem to be involved: (i) a representation of the geometry of spatiotemporal developments in the road traffic scene, comprising both a 2D-one in the image plane and a 3D-one relating to the depicted scene, (ii) a representation of driving maneuvers closely coupled to particular traffic situations, (iii) a conceptual representation of visible bodies, their attributes, and their elementary movements, (iv) generic conceptual representations of spatiotemporal body configurations and their expected temporal developments, and (v) one or more versions of a natural language representation of developments centred around the current point in time.

In other words, the goal of HN00 was to look at a 2D image of an intersection, from it construct a 2D/3D representation of the scene. In that scene identify and label objects and describe them in a conceptual language called SIT++. Once in that conceptual representation (as situation trees) they could conduct logical inference (using Fuzzy Metric Temporal Horn Logic) on their representation in order to decide what the agents they identified are trying to do.

Note that HN00 involved no natural language processing (NLP) at all. Although they did have to use lots of pattern recognition and various machine learning algorithm that would be familiar to NLP practitioners. However, their domain was transforming a visual scene into a conceptual (not natural language) internal representation.

How did AGN08 extend beyond this? They changed what they wanted to do. Their task was not simply to view a scene and transform it into an internal representation, but to then output that internal representation in a natural language description. Thus, they were adding a natural language generation system to HN00. Generating natural language from an internal representation is obviously an important part of NLP.

In the process of adding this functionality, AGN08 had to extend the internal representation in several ways. This was due to the fact that more internal information was required to generate good natural language output, and because they wanted to deal with more complex scenes than HN00. The paper focuses on this aspect of the work (extending the internal representation) and only tangentially touches on natural language output. They go into detail of the natural language output in:

R. Gerber, Naturlichsprachliche Beschreibung von Straßenverkehrsszenen durch Bildfolgenauswertung. Dissertation,Fakultat fur Informatik der Universitat Karlsruhe (TH), Karlsruhe, January 2000

Unfortunately, I am not willing to learn German and read a whole thesis in order to give you a more complete answer about the details. Before you try to do that yourself (hopefully you already know German) or look into more recent papers, I recommend learning some basics of NLP. A good source is the following question:

Looking for a good beginners reference to learn computational linguistics

share|improve this answer
Thank you for taking the trouble in explaining in such detail.As I assume, these papers are diffucult to implement for my task and are not related to what I am interested in. Would you suggest,in general how should I start or what should be the starting point and steps in building a surveillance system for situational assessment based on natural language. – chk Jun 11 '12 at 18:48

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.