Publication | ACM Symposium on User Interface Software & Technology 2012

Waken

Reverse Engineering Usage Information and Interface Structure from Software Videos

Abstract

Waken: Reverse Engineering Usage Information and Interface Structure from Software Videos

Nikola Banovic, Tovi Grossman, Justin Matejka, George Fitzmaurice

ACM Symposium on User Interface Software & Technology 2012

We explore the possibilities and opportunities related to reverse engineering usage information from screen-captured application video tutorials. We develop an application-independent system that recognizes UI components and activities, such as cursor movements and icon clicks, from an input set of video tutorials, We then present Waken, an enhanced video player, which showcases some of the design opportunities the are introduced by having this additional meta-data. In particular, users can directly interact with UI components in the video, such as icons and menus, to display associated information, or navigate to relevant moments in other videos. Initial results suggest that the system can successfully reconstruct many aspects of a UI without any prior application-dependent knowledge.

Download publication

Related Resources

Publication

2023

Neural Shape Diameter Function for Efficient Mesh Segmentation

Introducing a neural approximation of the Shape Diameter Function,…

Article

2023

Advancing Construction Processes with Industry Collaboration

Learn how Autodesk Research and Howick are collaborating to push the…

Publication

2010

Towards Coherent Image Space Stylization of Animated 3D Shapes

We describe a rendering technique for creating animations of 3D shapes…

Publication

2021

Collective Transport of Unconstrained Objects via Implicit Coordination and Adaptive Compliance

We present a decentralized control algorithm for robots to aid in…

Get in touch

Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.

Contact us