Full Paper at EGEV 2020

In May 2020 the University of Norrköping hosted the joint conferences, Eurographics & Eurovis 2020. This was the first time the two conferences have been run as a single event and due to the global pandemic situation it was held virtually. Nevetheless the conference has been a great success.

We were pleased that one of the papers from our successful collaboration with Bournemouth University was invited for presentation at the conference. Long Chen presented our learning-based framework for semantic-level interaction, which has application in context-aware mixed reality.

Flowchart of the Framework
Figure 1: Flowchart of the Framework [JPG]

The framework is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. Our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.

A mixed reality throwing plates game. The interaction is different when throwing plates to (a) book, (b) desks, (c) computer screen and (d) chair.
Figure 2: A mixed reality throwing plates game. The interaction is different when throwing plates to (a) book, (b) desks, (c) computer screen and (d) chair. [JPG]

Reference

  1. Chen, L., Tang, W., John, N. W., Wan, T. R., & Zhang, J. J. (2020). Context-Aware Mixed Reality: A Learning-based Framework for Semantic-level Interaction. Computer Graphics Forum, 39(1), 484–496. [bib]