Li et al. 2023. Swarm Manipulation in Virtual Reality. ACM SUI '23.

Swarm Manipulation (SUI'23)

This paper introduces Swarm Manipulation, a VR-based technique for swarm control, and compares its performance to Virtual Hand and Controller methods through a 12-participant study. The evaluation spans three tasks across five conditions, revealing Swarm Manipulation's faster speeds in most tasks and reduced resizing errors. However, a speed-accuracy trade-off exists in rotation tasks. These findings suggest the technique's potential to improve VR usability and set the stage for future research on swarm interaction and control.


Gao and Li et al. 2023. VR PreM+: An Immersive Pre-learning Branching Visualization System for Museum Tours. Chinese CHI '23.

VR PreM+ System (Chinese CHI'23)

We present VR PreM+, an innovative VR system enhancing web exploration with immersive 3D environments. Using keyword-based information retrieval, it improves content management and comparison. Our studies show efficient info retrieval, heightened engagement, and presence, guiding future VR info systems. VR PreM+ bridges traditional web browsing and immersive VR for interactive information acquisition, with applications in research and education.


Li et al. 2023. Creating On-Body Menus in Virtual Reality. Under Review.

Creating On-body Menus

This paper explores the impact of creation processes on the effectiveness and user experience of on-body menus in virtual reality (VR). On-body menus are virtual interfaces situated directly on the user's body in the VR environment. A user study (N=12) compared three creation processes (first-person, third-person, and mirror perspectives) in terms of performance, user experience, and preferences for organizing applications on the body. The Mirror perspective had the best recall time and accuracy for memory recall, and specific categories of applications were consistently placed on different body areas. Design strategies for on-body menus in VR are provided based on the findings. The study emphasizes the importance of considering different creation processes for designing effective on-body menus in VR, offering valuable insights for designers and developers.

Under Review

Patibanda et al. 2023. Fused Spectatorship: Designing Bodily Experiences Where Spectators Become Players. PACM HCI: CHI PLAY 2023.

Fused Spectatorship (CHI PLAY '23)

We propose "Fused Spectatorship", where spectators control their hands using an EMS system to actively participate in digital games. A study showed participants couldn't distinguish between watching their hands play and playing the games themselves. We offer design considerations and discuss ethical implications for future fused spectatorship experiences.

Video | Presentation | DOI | PDF

Patibanda et al. 2023. Auto-Paízo Games: Towards Understanding the Design of Games that Aim to Unify a Player’s Physical Body and the Virtual World. PACM HCI: CHI PLAY 2023.

Auto-Paízo Games (CHI PLAY '23)

We propose "Body as a Play Material", a novel approach that unifies the physical body and the virtual world in digital bodily games. By using an Electrical Muscle Stimulation (EMS) system, players control one hand (input) to compete against the other hand (output). Thematic analysis of a study with 12 participants revealed four player experience themes, highlighting engagement with bodily movements and the ambiguity of using the body as play material. Our work aims to bridge the divide between the physical and virtual realms.

Video | Presentation | DOI | PDF

Mueller et al. 2023. Towards Understanding the Design of Intertwined Human-computer Integrations. TOCHI 2023.

Intertwined Human-Computer Integrations (TOCHI '23)

Our work explores human-computer integration trends, specifically focusing on shared agency over the user's body between the user and computational machines. Termed "intertwined integration," this approach blurs the distinction between user and machine control. Through three case studies, we propose two dimensions ("awareness of machine's agency" and "alignment of machine's agency") to define a design space. We identify four roles (angel, butler, influencer, and adversary) that machines can assume within this space and discuss strategies for designers based on our experience. Our aim is to advance the understanding of shared agency in the HCI field.

Presentation | DOI | PDF

Mueller et al. 2021. Limited Control Over the Body as Intriguing Play Design Resource. CHI 2021.

Limited Control Over the Body (CHI '21)

The integration of interactive play and the human body in "bodily play" systems is gaining interest. While these systems prioritize player control over bodily actions, we explore the potential of "limited control over the body" as a design resource. Using three bodily play systems, we demonstrate how designers can engage players through varying degrees of indirect control. We propose four strategies: Exploration, Reflection, Learning, and Embracement, to employ limited control over the body. By applying these strategies, designers can enhance body engagement and promote the benefits of play.

Presentation | DOI | PDF

Lu et al. 2020. Exploration of Hands-free Text Entry Techniques For Virtual Reality. IEEE ISMAR 2020.

Hands-free Text Entry (ISMAR '20)

We propose "Fused Spectatorship", where spectators control their hands using an EMS system to actively participate in digital games. A study showed participants couldn't distinguish between watching their hands play and playing the games themselves. We offer design considerations and discuss ethical implications for future fused spectatorship experiences.

Presentation | PDF (DOI)

Xu et al. 2020. Exploring Visual Techniques for Boundary Awareness During Interaction in Augmented Reality Head-Mounted Displays. IEEE VR 2020.

Boundary Awareness  (IEEE VR '20🏅️)

We explore visual techniques for boundary awareness in Augmented Reality (AR) Head-Mounted Displays (HMDs) during mid-air hand interactions. Users often move their hands outside the tracked area, impacting dynamic tasks. Boundary awareness issues in AR devices are underexplored. Through a formative study, we identify challenges and propose four methods (static surfaces, dynamic surface(s), static coordinated lines, dynamic coordinate line(s)). We evaluate these methods against a baseline condition. Our findings show that visual boundary awareness methods can assist dynamic mid-air hand interactions, but their effectiveness depends on the user.

Presentation | PDF (DOI)

Xu et al. 2020. Results and Guidelines From a Repeated-Measures Design Experiment Comparing Standing and Seated Full-Body Gesture-Based Immersive Virtual Reality Exergames: Within-Subjects Evaluation. JMIR Serious Games (Q1, SCI). 

Standing vs. Seated Exergames (JMIR Serious Games)

This study compared full-body gesture-based iVR standing exergames with seated exergames. Among 52 participants, seated exergames showed higher heart rate, exertion, and perceived exertion. However, they also led to higher peripheral and sopite-related sickness. Seated exergames were rated higher in value/usefulness for intrinsic motivation. They offer benefits such as increased exertion and suitability for small spaces, but careful design and consideration of motion sickness are important.


Interactive Demos and Posters

Li et al. 2023. GesMessages: Using Mid-air Gestures to Manage Notifications. ACM SUI 2023. Poster.

GesMessages  (SUI '23)

This paper presents GesMessages, a mid-air interaction application for managing real-time message notifications on computing devices. Utilizing onboard cameras, the system supports three key gestures for notification management: expansion, hiding, and deletion. We detail the system architecture and discuss its applicability in context-aware systems, thereby contributing to the field of gestural interaction. This research advances our understanding of the role of mid-air gestures in message management and interactive systems.


Li et al. 2022. GesPlayer: Using Augmented Gestures to Empower Video Players. ACM ISS 2022. Poster.

GesPlayer  (ISS '22)

We introduce GesPlayer, a gesture-based video player that allows users to control playback, volume, and screen brightness through semantic gestures. By using hand gestures as an interface, users can navigate videos without the need for a mouse or keyboard. We aim to explore the inclusiveness of hand-based interaction and advance the use of semantic gestures in interactive environments through computational interaction methods.

Video | PDF (DOI)

Li et al. 2021. vrCAPTCHA: Exploring CAPTCHA Designs in Virtual Reality. ACM CHI 2021. Interactivity.


To address the need for exclusive and interactive CAPTCHA designs in virtual reality (VR) devices, this paper presents four traditional 2D CAPTCHAs (text-based, image-rotated, image-puzzled, and image-selected) adapted for VR. Additionally, two VR-specific CAPTCHA design prototypes (task-driven and bodily motion-based) are proposed based on VR's 3D interaction characteristics. A user study with six participants evaluated the feasibility of these designs. The findings suggest that the proposed VR CAPTCHAs can serve as inspiration for future CAPTCHA designs in VR.

Video | PDF (DOI)

Patibanda et al. 2021. Actuating Myself: Designing Hand-Games Incorporating Electrical Muscle Stimulation. ACM CHI PLAY 2021. WIP

Actuating Myself (CHI PLAY '21)

We explore the use of Electrical Muscle Stimulation (EMS) technology to enhance traditional hand-games like Rock-paper-scissors and Thumb-war. We designed and evaluated two EMS games, "Slap-me-if-you-can" and "3-4-5," which can be played individually. Through thematic analysis, we identified three key themes: gameplay experiences and the influence of EMS hardware, interaction with EMS and the calibration process, and shared control and its impact on playing EMS games. Our findings contribute to the understanding of EMS in supporting hand-games, advancing movement-based games as a whole.

Video | PDF (DOI)

Li et al. 2021. Myopic Bike and Say Hi: Games for Empathizing with The Myopic. ACM CHI PLAY 2021. Student Game Design Competition

Myopic Games (CHI PLAY '21)

Myopia, a prevalent eye condition, impacts people's ability to focus on distant objects and reduces their quality of life. To bridge the empathy gap, we developed two virtual reality (VR) games, "Myopic Bike" and "Say Hi," simulating daily challenges faced by myopic individuals. Through questionnaires and interviews with four participants, we found that the games successfully engaged non-myopic players and fostered empathy towards myopia sufferers. Our VR games offer an immersive and judgment-free experience, promoting understanding and empathy for myopia.

Video | PDF (DOI)

Xu et al. 2020. VirusBoxing: A HIIT-based VR Boxing Game. ACM CHI PLAY 2020. Student Game Design Competition

VirusBoxing (CHI PLAY '20)

We developed "VirusBoxing", a boxing-based VR exergame, to address the time constraints of regular exercise. By incorporating high-intensity interval training (HIIT) and empowering players with enhanced abilities, such as precise jabs at distant objects, VirusBoxing offers an efficient, effective, and enjoyable exercise experience. Our paper discusses the adaptation of HIIT protocols and gameplay features to promote physical activity and improve players' health and quality of life.

Video | PDF (DOI)