Tesseract

This paper presents Tesseract, a novel Worlds-in-Miniature-based VR system that allows users to search through space and time in spatial design recordings. Users can form queries using the Search Cube interface. Queries are retrieved as spatial clips that users can preview or re-watch in 1:1 scale. Tesseract provides four querying tools to enable searching spatial design recordings. Users can manipulate objects into the search cube to perform object search or define the behavior of recorded people such as their proximity to objects through proximity search, their viewpoints through viewpoint search, or their speech through voice search to retrieve interesting moments related to design activities.

Abstract

Tesseract: Querying Spatial Design Recordings by Manipulating Worlds in Miniature

Karthik Mahadevan, Qian Zhou, George Fitzmaurice, Tovi Grossman, Fraser Anderson

ACM SIGCHI Conference on Human Factors in Computing Systems 2023

New immersive 3D design tools enable the creation of spatial design recordings, capturing collaborative design activities. By reviewing captured spatial design sessions, which include user activities, workflows, and tool use, users can reflect on their own design processes, learn new workflows, and understand others’ design rationale. However, finding interesting moments in design activities can be challenging: they contain multimodal data (such as user motion and logged events) occurring over time which can be difficult to specify when searching, and are typically distributed over many sessions or recordings. We present Tesseract, a Worlds-in-Miniature-based system to expressively query VR spatial design recordings. Tesseract consists of the Search Cube interface acting as a centralized stage-to-search container, and four querying tools for specifying multimodal data to enable users to find interesting moments in past design activities. We studied ten participants who used Tesseract and found support for our miniature-based stage-to-search approach.