Active Detection of Visual Situations in Images

Active Detection of Visual Situations in Images
October 8, 2019

Speaker: Melanie Mitchell, PhD

Professor of Computer Science, Portland State Univ. and External Professor and Member of the Science Board at the Santa Fe Institute

Thursday, Oct 17th | 12:30-1:20pm | BANN 102

This talk describes an AI architecture for detecting visual situations in images. Visual situations areconcepts such as “a boxing match”, “a handshake”, or “a crowd waiting for a bus”, whose instantiations in images are often linked more by their common spatial and semantic structure than by low-level visual similarity. Given a query situation description, our architecture—called Situate—learns models capturing the visual features of expected objects and the expected spatial configuration of relationships among them. Given a new image, Situate uses these models in an attempt to instantiate (locate in the image) each expected component of the situation and computes a score indicating the degree to which the new image is judged to contain an instance of the situation. Such scores can be used to rank images in a collection. I will demonstrate the promise of this system by comparing its performance with that of a baseline method and a related system based on “scene graphs”.

Prof. Mitchell attended Brown University, where she majored in mathematics and did research in astronomy, and the University of Michigan, where she received a Ph.D. in computer science. Her dissertation, in collaboration with her advisor Douglas Hofstadter, was the development of Copycat, a computer program that makes analogies. Melanie has held faculty or professional positions at the University of Michigan, the Santa Fe Institute, Los Alamos National Laboratory, the OGI School of Science and Engineering, and Portland State University. She is the author or editor of six books and numerous scholarly papers in the fields of artificial intelligence, cognitive science, and complex systems. Melanie’s book Complexity: A Guided Tour (Oxford University Press) won the 2010 Phi Beta Kappa Science Book Award and was named by as one of the ten best science books of 2009. Melanie’s latest book, Artificial Intelligence: A Guide for Thinking Humans, will be published by Farrar, Straus, and Giroux in October, 2019. Melanie originated the Santa Fe Institute’s Complexity Explorer platform, which offers online courses and other educational resources related to the field of complex systems. Her online course “Introduction to Complexity” has been taken by over 25,000 students, and is one of Course Central’s “top fifty online courses of all time”.