Dr. Michael Blighe

Project - SenseCam work at Joanneum Research, Austria

Michael is developing an overall framework / data hierarchy with which to structure and organise a collection of SenseCam images. At the top level he will use location (GSM, GPS), then try and detect forms of social context (Bluetooth, accelerometer) and finally perform some clustering at the lowest levels using low-level features and other metadata (SenseCam metadata, temporal, etc.).

In Austria Michael focuses on this last issue - clustering. He is using the Scale Invariant Feature Transform (SIFT) features. These features are invariant to changes in image scale, rotation, intensity and to moderate affine transformations. This makes them useful for detecting similar "image patches" or regions across multiple images where the matching "image patch" may have changed due to any of the issues above (scale, etc.). This is a regular feature in SenseCam images due to the movements of the user wearing the camera and you can see examples of this in Slides 4-6. By extracting these features, he hopes to generate an image "signature" and then cluster the images based on this "signature" (using the Xmeans algorithm to generate the signature). The idea is that the long sequences of very similar images you get with a SenseCam during a typical day (e.g. when you're at your desk all day or watching TV in the evening) would be clustered together. Images where there is more rapid movement should not have many matching SIFT features. So, it's a method to attempt to cluster SenseCam images which occur in the same "setting".

Finally, as an aside, using SIFT features also allows the exploration of object detection in SenseCam images. This has not been explored as yet, but in theory it should be possible to generate SIFT features for an image of a particular object (say your computer monitor) and then compare that SIFT feature vector to those stored in your database. In theory, it should find all SenseCam images containing a similar computer monitor - no matter what scale, what rotation, etc. So, this could be a very interesting area for future exploration. Slides

Dr. Georgina Gaughan

Project - The detection of features within SenseCam images

The detection of semantic information from within a video sequence is very important. The automatic extraction of the semantic meanings from a visual image or video sequence is, however, a highly complex task. The area of concept detection has become a hot research topic over the last decade however major challenges still remain namely accuracy and coverage. However, the accuracy of these feature/concept detectors can be very low and in many cases the inclusion of such feature evidences will inevitably degrade the accuracy of a retrieval system. To date many concept detectors have been developed for specific domains such as the detection of a goal being scored in a soccer game, and a limited set of concepts have been developed by groups to detect a certain concept in video data such as the ``beach'', the ``sky'' or the presence of a named individual. DCU in collaboration with MediaMill, located in the Netherlands are exploring the issues of identifying features within the SenseCam images. This is a particularly challenging issue as SenseCam images are of a very low resolution. We are currently investigating the presence of 101 features on a collection of 1000 SenseCam images which contain various features using concept detectors developed by MediaMill. Preliminary results suggest that it is indeed possible to accurately detect some features such as indoor, outdoor, sky, grass and people walking. This is an ongoing research topic.

Barry Lavelle

Project - SenseCam & Bluetooth

Our SenseCam research will focus on social event detection using Bluetooth. Preliminary research has indicated there is a strong correlation between the levels of Bluetooth activity encountered and the presence of people. By combining this work with face-detection, we will attempt to avoid cases where false-positives may occur e.g. the wearer enters a lab in which many Bluetooth devices are present but the owners' are all out to lunch.

Further extensions of our research on familiarity and social networks will allow events to be classified by the type of event. This will be based on entropy levels determined through the user's familiarity of the Bluetooth devices encountered for a particular event. For example, encountering a high number of devices with a low familiarity score would suggest a salient event for the user and vice-versa.