My contribution to the Information Exploration workshop will be on the design of interfaces for multimedia organization and retrieval. I will approach this from the perspective of consumer home media applications, such as digital photography and videography. I will discuss information representation, query and search techniques, content-based indexing/retrieval, and user interface considerations from the standpoint of the emerging consumer market for digital media.

The rapid growth of the consumer digital media market will give rise to difficulties for consumers in finding items of interest in their collections. To make matters worse, market research has found consumers to be resistant to the notion of organizing and managing home media, seeing these activities as tedious. Photos tend to be thrown in a shoebox; home videos sit on shelves unviewed.

Technologies for multimedia organization and retrieval have been applied to problems in business and professional imaging, with some success. However, it is not clear that these solutions will work as well for consumer-oriented applications. Traditional keyword-based search technologies are very effective but their use for visual media require manual entry of textual annotations. Technologies for content-based indexing and retrieval provide some degree of automation for this process, but depend upon low-level perceptual features, which are not as meaningful to consumers. Finally, much research attention has been paid to the task of direct search, but relatively minor attention has been given to the activities of browsing that are more likely to be employed by consumers than direct search.

Our project at HP Labs is experimenting with hybrid approaches to these issues. First, we have built prototype systems which make it easier to manually annotate while utilizing automated methods where appropriate. Second, we are looking at methodologies for effective "metadata" representation, to model the user-level concepts and interrelationships that are appropriate for describing and indexing digital media objects. Third, we are experimenting with techniques for browsing and visualizing large digital media
collections. This third area is my area of personal responsibility on the project and is in line with my long-term professional interests in information sharing in collaborative multimedia applications(for more background on this topic, see Nardi et al, 1995).

I am particularly interested in how to optimally use both peoples' cognitive and perceptual capabilities in browsing/visualization interfaces. I am also interested in how to balance the tradeoffs between automated feature extraction, which is fairly perceptually-oriented, and human indexing/annotation, which is primarily cognitive in nature. Whereas many people argue that the most effective visualization interfaces result from shifting as much activity as possible from cognitive to perceptual domains, I believe we have not yet exhausted the possibilities for building upon peoples' cognitive abilities.

In my work related to browsing/visualization, I have collaborated with artist Stuart Cudlitz (Colossal Pictures and Media Concrete, San Francisco), who has applied J.J. Gibson's concept of "gradients" as a structuring principle for his design of multimedia artworks. He has claimed that clustering visual images along a "content gradient" (Cudlitz 1995) evokes the perceptual abilities of the viewer and allows the viewer to make sense of larger amounts of information.

I have worked with Cudlitz to apply his notion of a "content gradient" to develo a computational model and prototype application for browsing collections of media objects. Implicit in this computational model is the assumption that such "content gradients" can be generated from the strengths of features extracted bycontent-based indexing tools. For example, size and prominence of a recognized face in an image can be used to define a "strength" for a given attribute in a gradient. In other words, photos of a specific person could be arranged along a gradient sorted by decreasing prominence of that person's face in each photograph.

The prototype has received positive reactions from consumers in focus groups but a number of conceptual and technical issues remain. First, the meaningful information used by people in organizing/retrieving materials is semantic, more cognitive in nature. Is it plausible to arrange such higher-level information into a quantitative gradient or even into an ordinal structure? At the same time, automated feature extraction methods provide quantitative measures that lend themselves well to the building of gradient structures. But does it follow that these gradient structures will really provide perceptual cues that enable users to make sense of larger amounts of information?

During the workshop, I plan to describe our work and discuss these issues -- of cognitive vs. perceptual factors, of automated vs. human annotation -- as they relate to the construction of effective interfaces for multimedia information exploration.

Biographical Information:

Allan Kuchinsky is a Principal Project Scientist at Hewlett Packard Laboratories in Palo Alto, California. His current work focuses upon multimedia organization and retrieval technologies. Previous work includes application architectures for broadband residential information systems, collaborative multimedia systems, active electronic mail systems, artificial intelligence tools, and VLSI circuit design technologies. Allan serves as Co-Chair for the ACM Special Interest Group on Multimedia (SIGMM).

Publications relevant to this workshop:

Nardi, B., Kuchinsky, A., Whittaker, S., Leichner, R., and Schwarz, H., "Video as Data: Technical and Social Implications of a Collaborative Multimedia Application", Journal of Computer-Supported Cooperative Work, Fall, 1995, and in Wilbur, S., Finn, K., and Sellen, A., eds, "Video Mediated Communication", Lawrence Erlbaum Associates, 1996.

Kuchinsky, A., "Bit Velocity is Not Enough: Content and Service Issues for Broadband Residential Information Services", in Proceedings of the IEEE 3rd International Workshop on Community Networking, Antwerp, Belgium, May, 1996.

Nardi, B., Schwarz, H., Kuchinsky, A., Leichner, R., and Whittaker, S., "Turning Away from Talking Heads: The Use of Video-as-Data in Neurosurgery", Proceedings of InterCHI'93, Amsterdam, April 1993, and in Emmott, S. "Information Superhighways: Multimedia Users and Futures ", Academic Press, London, 1995.

Sclabassi, R., Leichner, R., Kuchinsky, A., Krieger, D., and Prince, F., "The Multi-Media Medical Monitoring, Diagnosis, and Consultation Project", Proceedings of Hawaii International Conference on Systems Sciences, Jan 1991.