Syntharch: interactive image search with attribute-conditioned synthesis

[摘要] The use of interactive systems has been proposed and found to be a promising approach for content-based image retrieval, the task of retrieving a specific image from a database based on its content. These systems allow the user to refine the set of results iteratively until the target is reached. In order to proceed with the search efficiently, conventional methods rely on some shared knowledge between the user and the system, such as semantic visual attributes of the images. Those approaches demand the images to be semantically labeled and introduce a new semantic gap between the two parties’ understanding. In my thesis, I explore an alternative approach to interactive image search where feedback is elicited exclusively in visual forms, therefore eliminating the semantic gap and allowing for a generalized version of the method to operate on unlabeled databases.Thanks to the recent advancements in generative adversarial networks, we can now generate realistic images of certain controlled characteristics and use a multidimensional attribute space learned from an image database to condition image synthesis. I present Syntharch, a novel interactive image search approach which uses synthesized images as options instead of textual questions to gain information on the relative attribute values of the target image. For each iteration of the search, rather than asking the user to make an attribute-value comparison in words, Syntharch generates a pair of options (synthesized images) which varies only in one attribute and let the user select the option that is more visually similar to the target.I then demonstrate that using synthesized images rather than real images retrieved from the database as feedback options, Syntharch causes less confusion to the user. Further, I establish that the specific search method I propose performs similarly or better in comparison to the conventional approach.Overall, my thesis presents a new approach of interactive image search, proposes a specific implementation following that approach, and validates the hypotheses that guided the search approach as well as the implementation choices.

[发布机构] the University of Pittsburgh