Touch Projector Mobile Interaction Through Video

Touch Projector: Mobile Interaction through Video [1]


The Touch Projector uses a mobile phone with video camera to allow the user to interact with remote displays in view of the camera. The system requires special software not only on the mobile phone, but also on each of the systems controlling the remote displays. The paper describes three extensions to an earlier design.


  1. The technique allows interaction with displays that are not necessarily reachable (e.g., on the other side of a glass window) or do not have associated interaction hardware (e.g., a mouse or touch screen). Some of the ideas in the paper are directly applicable to interaction with a large video wall, such as in the Lawson Commons.
  2. The technique of auto-zooming the display simplifies locating interaction points on the remote display.
  3. The technique of freezing the phone display helps compensate for problems caused by unsteady hands.
  4. The paper includes a large collection of references to related work, including augmented reality and "world in miniature".
  5. The paper includes useful performance statistics, e.g., that the iPhone 3GS is limited to about 8 fps of video.
  6. The research includes a detailed user study, using a within-subjects experimental setup.


  1. In the end and for some situations, it may be easier and more robust to just use a Remote Desktop protocol. In fact, they suggest that future work includes use of a computer-generated mobile screen rather than relying on camera streaming.
1. Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, Patrick Baudisch, Touch Projector: Mobile Interaction through Video.