Crossmodal Search and Visual Exploration of 3D CH Objects

3D objects are used in many application areas to represent form, appearance and function of natural and man-made objects. Important usages of 3D objects are e.g., mechanical engineering, where parts and components are represented by 3D objects, or photorealistic rendering, where 3D objects are used to model scenes of interest. Also, they are an important documentation type in the area of museums and digital libraries, where 3D objects represent object collections, archaeological artifacts, architectures etc. for presentation, research and long-term preservation. Novel methods for 3D digitization, interactive modeling and shape synthesis support the creation of increasingly large 3D object repositories. While for smaller repositories the individual inspection of objects by users is possible, this is not efficiently possible anymore for large repositories consisting of many objects. Rather, appropriate techniques for similarity search based on example objects or sketches are an essential functionality for exploring and leveraging large repositories. Existing methods for 3D search allow to search for globally or partially similar shapes, based on example search objects or sketches, and support the basic search in generic 3D objects. In this project, we will develop a new 3D search method for incomplete 3D query objects, as needed e.g., when working with cultural heritage (CH) object repositories in archaeological research. In this area, objects are often incomplete as artifacts over time have become broken or eroded, and hence are only partially given. Our basic idea is to enable users to interactively complete an incomplete 3D query shape by a simple sketch editor. Hence, the user is enabled to visually express a hypothesis for a possible shape, and a novel search algorithm will support the search for such hybrid queries consisting of a 3D object and a sketch extension. We will also research a new method for finding images or sketches contained in documents, similar in view to a 3D query object. This novel search method will support users by automatically cross-referencing 3D shapes to publications, providing context and comparison for research in large document repositories. Furthermore, we will research a new method for interactive exploration of search results by means of shape- and document visualization, supporting the comparison of search results. We will apply and evaluate these methods for effectiveness on relevant Archaeological 3D object and document data, together with research partners including the Institute of Archaeology at University of Graz. This project will contribute new approaches to 3D similarity search research and visual result exploration. It will also provide a new enabling methodology for search-based exploration and comparison of CH objects of relevance for Archaeological research.

The research project Crossmodal Search and Visual Exploration of 3D Cultural Heritage Objects was supported by the Austrian Science Fund FWF in co-support with the State of Styria. The goal of this project was to research novel methods for interactive search and visual analysis of collections of 3D cultural heritage objects, supporting the archaeological research process by innovative tools and approaches. This project was carried out together with the Institute of Classics at University of Graz, and the Computer Science Department at the University of Chile, Santiago de Chile. The key question of the project was how domain scientists can interactively search, compare and analyze the shapes, paintings, and ornamentation of cultural heritage objects, based on 3D object and 2D image representations. The project achieved rich results, including several novel approaches for interactive and sketch-based visual search and comparison of objects. The approaches are based on effective user interfaces and data visualization. They were evaluated quantitatively and qualitatively, including use case studies together with domain scientists, and based on existing and newly created data benchmarks. In addition, 2D and 3D digitization was conducted to arrive at representative test data, which in turn inspired the research. The results are documented in journal and conference proceeding publications, talks, and the organization of a workshop. The project enabled the dissertation of the key PhD candidate project researcher, and supported the scientific training of an undergraduate and a graduate research student. In addition, a Master thesis was written in this project. To summarize, the project has achieved all its set goals, and inspired follow-up research.