Introduction
This project builds upon a previous application I developed under the same name, which aimed to explore the potential of using virtual reality to facilitate ballroom dance learning. To streamline the scope, the original project focused exclusively on International Rumba, with the idea that success could lead to expansion into other dance styles. The initial phase centered on creating two distinct modes: a class-like structure for learning the fundamentals and a freestyle mode to practice the material in a more flexible setting. While the project offered valuable preliminary insights, it did not yield definitive results.
Due to the last project resulting in only a partially complete application, I made the decision to continue with it's development in hopes of crafting a more complete experience that would allow for a detailed analysis of the merit of the game. In particular, the goals of the project included:
To create a course sequence for both lead and follow
To create a scoring system that would inform players how well they did
To create a simple story line
Additional goals were also considered but were scoped out due to time constraints. These additional features included
A new environment that players could dance in
A competition system for providing a benchmark of how well players are doing
Despite the challenges encountered, the project came together to deliver a satisfying experience. Overall, I believe the project has potential, particularly as a tool for viewing and learning lessons. However, its utility as a platform for dancing with a virtual partner is more limited, as will be explained in further detail.
Development
Free Style System Overhaul
The project began with a comprehensive overhaul of the original freestyle system. Initially, the system was designed to randomly cue up dance moves and transition through them consecutively. The intention was to help users quickly recall different movements. However, the system fell short in terms of usability and functionality, prompting several significant changes, as outlined below.
For starters, the original system lacked an effective way to signal what move was coming next, an issue since, in real-world dancing, a partner connection typically provides this information. Given the project's time constraints, creating a virtual equivalent of this connection was not feasible. Instead, two intermediate solutions were developed:
Visual Display: A glanceable visual display was added to the user interface, showing upcoming moves. This approach, inspired by Just Dance, provided visual cues akin to the dance position indicators at the bottom of their screen.
Announcer System: An audio-based announcer was implemented to notify the user of the next move a few beats before the transition. This feature allowed users time to prepare for the motion.
Although these solutions mitigated the issue, they were not perfect, as both added distractions that detracted from the actual dancing experience. However, for a prototype, they effectively addressed the problem enough to evaluate the viability of the system.
Additionally, to enhance user interaction, a feedback system utilizing haptics and scoring was introduced:
Haptic Zones: Haptic zones were added to the virtual dance partner. Bubbles appeared in specific areas where users needed to place their hands. When users maintained their hand in the correct zone, a corresponding haptic response was triggered on their controller.
Scoring System: Users earned points for every beat they maintained their hand in the correct zone. At the end of the session, a final score provided a summary of their performance.
This system served two purposes: immediate feedback on hand positioning and longer-term performance evaluation. It also acted as a visual aid, guiding users to position their hands correctly for each move.
Finally, the freestyle mode was rebranded as Practice Mode to better align with its purpose. The original concept aimed to encourage creativity and improvisation, but implementing such a feature proved challenging in practice. The focus was shifted toward creating a structured review mode to complement the lessons taught in the application's main storyline.
Overall, these changes collectively improved the system's functionality, making it more practical for its intended purpose while laying the groundwork for potential future expansions.
Animation System
The animation system was the next critical area that needed improvement to bring the app together. Initially, the application relied on motion capture recordings of my dance partner and me, captured using the trackerless AR51 system. However, during the first phase of testing, it became clear that these animations had several issues. The captured motions often exhibited problems such as feet floating off the ground or arms moving unnaturally. Furthermore, these animations were originally placeholders and were not always synchronized with the music, as there had not been an opportunity to recapture them during the first phase. Transitioning between moves also proved problematic, with blending causing unrealistic shifts in the dancer's position, leading to significant movement and sliding during transitions. Finally, the system's animation tree made it difficult to add new motions, as each move had to be carefully linked and blended to avoid these sliding issues.
To address the first issue, my dance partner and I scheduled a session with a tracker-based Vicon Motion Capture System located in my graduate studio. This system allowed us to capture higher-quality animations, ensuring that all movements began on the first beat and were performed at a fixed tempo. The fixed tempo could then be adjusted later by speeding up or slowing down the animations at consistent rates. This upgrade immediately improved the experience, making it feel more like dancing with a real partner.
To resolve the second issue, the animation system was redesigned to incorporate animation layers, which enabled multiple animations to run simultaneously for different body parts. This separation allowed the root position transitions to remain unaffected by blending, while the rest of the body retained some blending to create the illusion of connected movements. While not entirely realistic, this solution significantly stabilized the animations during virtual partner dancing.
To tackle the final issue, I redesigned the code to manually handle transitions between animation states. This approach allowed for dynamic control over an expanding roster of moves, replicating the functionality of the Unity Animator while offering far greater flexibility. By bypassing the rigid structure of the animation tree, it became much easier to integrate new animations, enabling the system to grow and adapt without sacrificing usability.
Overall, these updates to the animation system significantly refined the user experience. The app now feels more polished and responsive, and the improved framework makes future modifications far simpler while maintaining high-quality performance. .
Story Mode
The final major update was the incorporation of a story mode, driven by my interest in narrative-driven VR games. I believed that adding a story would not only enhance the user experience but also motivate players to stay engaged with the app. Furthermore, the story mode provided a structured framework to present a course sequence, helping users internalize movements and knowledge through repeated practice within a meaningful context.
The story mode consisted of three key components: cutscenes, lessons, and practice sessions. This approach allowed the narrative to unfold through cutscenes while lessons introduced material in a logical progression. Practice sessions then reinforced the material from previous lessons using the updated practice mode, which was now restricted to moves that users had already learned. This structure ensured that users could solidify their skills incrementally.
The story mode followed a clear progression:
Starting Cutscene – Introduced the narrative context and set the stage for learning.
Frame Lesson – Taught how you stand and hold your frame in rumba
Basic Lesson – Introduced the first steps of the rumba that the rest would build upon
Practice Session – Introduce the idea that after each lesson a review phase would begin
Repeated for additional lessons and practices:
New Yorker Lesson → Practice
Shoulder to Shoulder Lesson → Practice
Cucaracha Lesson → Practice
Spot Turn Lesson → Practice
Final Cutscene – Concluded the narrative and provided a sense of closure.
Main Menu Redesign
Additionally, with the inclusion of a story mode, the main menu underwent a significant redesign. The traditional 2D interface was replaced with an immersive 3D environment: the dance studio where the practice sessions took place. In this redesigned menu, characters representing different menu options stood in various parts of the studio that you would be able to walk around and interact with:
Lead Instructor: Allowed users to resume the story.
Follow Instructor: Provided access to replay lessons.
Dance Partner: Served as the entry point to practice mode.
Players could interact with these characters by approaching them and initiating a conversation, seamlessly blending the interface into the immersive environment.
Overall, these updates helped create a more intentional and cohesive experience. The story mode added narrative depth and a sense of progression, while the redesigned menu offered clear direction on what players needed to do next. Together, these changes transformed the app into a more engaging and user-friendly platform for learning ballroom dance.
Lesson Structure
A minor but impactful change involved simplifying the system for executing lessons. The code was refactored to streamline the creation of new lessons, driven by the decision to simplify the overall lesson structure. Previously, the lessons followed a lengthy sequence:
Intro → Lesson → Practice Individually (½ speed) → Practice Individually (full speed) → Practice with Partner (½ speed) → Practice with Partner (full speed) → Outro.
This structure was revised to tighten the experience and make it more user-friendly. The individual practice steps were removed and incorporated directly into the main lesson, while the intro and outro sections were eliminated entirely. The new streamlined structure ensures that the focus remains on the most essential aspects of the learning process, minimizing unnecessary downtime for the user.
Additionally, like the animation system, the lessons from the first phase were originally intended as placeholders. To enhance the educational value, I rerecorded each lesson with greater depth and detail, creating a more thorough and engaging learning experience for users.
These changes not only improved the lesson flow but also made it easier to develop and expand the system, ensuring a more polished and efficient experience for future updates.
User Testing and Refinement
After implementing the developments described above, I tested the experience with my dance partner and several inexperienced dancers to identify strengths and areas needing improvement. The feedback received highlighted three key issues, summarized below.
Haptic System Feedback Was Unintuitive
The initial haptic system did not effectively simulate the feeling of dancing with a partner, as the feedback provided was unclear and counterintuitive. Testers suggested inverting the haptic response: instead of triggering haptics when users were correctly aligned, the system would activate haptics only when the user’s hand was misaligned. This adjustment better mimicked the real-world sensation of pressure indicating misalignment during partnered dancing. After implementing this change, subsequent testing showed a significant improvement in user understanding and overall clarity of the system.
Dance Partner Starting Prematurely
Users found that the virtual dance partner often started dancing before they felt ready. This issue stemmed from two main problems:
The dance partner would sometimes spawn in awkward or inaccessible positions relative to the user.
The partner would automatically begin dancing after a fixed 4-beat countdown, regardless of the user’s readiness.
To resolve the first problem, the spawning system was redesigned to ensure the partner always appeared in the same accessible position relative to the user. To address the second issue, the system was modified so that the partner would not begin dancing until the user successfully connected with them, at which point the countdown would start. These changes made the experience more user-friendly and ensured users felt prepared before beginning each session.
Difficulty Bridging Instruction and Partner Dancing
Participants felt there was a steep jump between the lesson and dancing with the partner, making the transition challenging. To bridge this gap:
Confirmation screens were reintroduced between steps to allow users to take a moment before progressing.
Half-tempo and three-quarter-tempo practice sessions with the partner were added to ease users into the movements gradually before transitioning to full-speed practice.
These features provided a more gradual learning curve, helping users feel confident in their movements before tackling the full-speed dance.
Impact of User Testing
Through this stage of testing, the system became significantly more accommodating and user-friendly. Most of the feedback centered around user experience, which was expected at this phase of development. The adjustments made based on this feedback resulted in a final product that felt much closer to a polished, full-scale application.
By testing with both experienced and inexperienced dancers, I gained a clearer understanding of how viable this project could be as a method for teaching partnered dancing. The improvements not only enhanced usability but also highlighted the potential for further development in making virtual reality an effective and engaging medium for learning ballroom dance.
Issues and Future Research
In short, I am pretty proud with how the application came out in the end due to, in my eyes, being a decently effective method of providing virtual lessons. Although, I do think that there are several limitations holding it back from being a viable app in the long term. Below is a list of the limitations discovered
Limitations in Floor Size
The first issue, limitations of floor size, stems from the spatial requirements of ballroom dancing. Over half of the standard ballroom dances require a large area to practice effectively. Specifically, a minimum of 9 by 12 feet is recommended to practice anything beyond basic steps. As a result, the virtual experience is limited to only 8 out of the 19 recognized ballroom dances, as the others demand significantly more space for proper execution.
Lack of Ability to Communicate Connection
The second issue, lack of ability to communicate connection, highlights a core challenge in translating ballroom dancing into a virtual format. Connection, a critical aspect of following in ballroom dance, relies on the lead dynamically conveying movement intentions through pressure changes, typically via hand-to-hand contact. This nuanced feedback allows the follower to react fluidly to the lead’s movements. Currently, the only viable method to simulate connection in virtual reality is through haptics. However, haptics generally operate as a binary on/off system, making it difficult to replicate the subtle variations in pressure needed to communicate effectively. Addressing this limitation will require careful consideration and further exploration of alternative approaches to simulate connection in a meaningful way.
Need For a Solution to Link Together Moves
The third issue, the need for a new solution to link together moves, highlights the challenge of creating smoother and more realistic transitions between movements. Currently, transitions between positions—especially in arm movements—are too rapid, leaving little time for the user to react. In traditional practice, this issue is mitigated by the connection between partners, which provides real-time feedback and guidance. However, as noted in the second issue, this connection cannot be relied upon in the virtual environment.
To address this challenge, two potential approaches have been identified:
Manipulating Motion Matching:
Motion matching is a technique that dynamically connects motions, potentially making transitions more seamless. Its strength lies in automatically blending animations based on context. However, motion matching is primarily designed for traveling motions, where a character moves across space, rather than for in-place movements typical of ballroom dance. This limitation makes it less effective for this specific use case.
Manually Adding Transitions:
Another approach is to manually design and implement transitions between each move. This would ensure more realistic and tailored connections, allowing for smoother transitions that better reflect ballroom dance techniques. However, this method requires considerable time and effort, as each transition would need to be carefully crafted and flexible enough to accommodate dynamic selection of moves.
Both options present trade-offs. Motion matching could reduce development time but may not fully meet the needs of in-place dance movements. Manual transitions, while more precise, would significantly increase the workload and development complexity. Further exploration is needed to determine the best path forward.
Effective Determination of User Struggles
The fourth issue revolves around the significant challenge of accurately determining what a user is struggling with during their practice. The lack of foot tracking capabilities meant that many common errors made by new dancers were undetectable, leading to gaps in the feedback provided by the application.
Specifically:
Users often stepped with the wrong foot, disrupting the rhythm and flow of the dance.
Feet were sometimes placed in unstable positions, which could hinder balance and movement.
Some users neglected to move their feet entirely, limiting their ability to execute proper steps.
Additionally, the scoring system relied on an assumed arm position for the user to function correctly. While theoretically sound, this approach fell short in practice because real-life dancers frequently adjust their hold to accommodate differences in height and body shape. These shortcomings cumulatively hindered the application's ability to provide meaningful and actionable feedback, which is essential for effective training.
To address these challenges, several improvements could be explored:
Foot Tracking Technology:
Incorporating advanced foot tracking would enable the application to monitor users' foot movements accurately. This would allow for real-time feedback on step accuracy, stability, and adherence to proper techniques, significantly enhancing the training experience.
Calibration Mode:
A calibration feature could guide users in finding a comfortable and functional hold with their virtual partner. This customization would ensure the hold is both user-specific and useful for maintaining proper technique, improving overall adaptability and realism.
Diverse Virtual Partners:
Introducing virtual dance partners with varying heights and body shapes would allow users to practice adjusting their hold dynamically, replicating the variability of real-world dancing. This addition would help users build the skills necessary to dance with a wide range of partners.
Addressing these issues would not only improve the application's ability to provide detailed feedback but also enhance its value as a training tool for dancers at all levels. These solutions emphasize the importance of realistic interaction and adaptability, both of which are crucial for creating an effective and immersive learning experience.
Showing AI Dancing Together
The final issue identified was the challenge of simulating dancing between different dance partners, particularly in the context of implementing a competition simulator. This feature was originally envisioned to provide feedback to the user and serve as a benchmark for progress by showcasing other dancers on the floor. However, during development, a significant problem emerged: the virtual dancers became increasingly offset from one another as the dance progressed.
This issue was traced to the fact that the individual dance moves did not start and stop in the same position. As a result, each subsequent move began from a slightly incorrect position, causing cumulative offsets between the dancers over time. This misalignment became more pronounced as the sequence continued, breaking the realism and fluidity of the simulation.
Fortunately, this issue is fundamentally similar to the problem of linking movements together, as described in the third issue. The same solutions being explored for improving transitions between moves—such as motion matching or manually adding transitions—should also apply here. By ensuring that movements are properly aligned and transitions are seamless, the offset problem in the competition simulator could likely be resolved as well.
Addressing this issue is crucial for creating a polished competition feature, as the ability to realistically simulate multiple dancers performing together is essential for maintaining immersion and providing meaningful feedback to users.
Outcomes
While the list of successes may initially seem modest compared to the challenges faced, they reveal the promising potential of a ballroom dance game. Some of the key achievements are outlined below:
Increased Interest in Ballroom Dancing
One of the most notable successes was the increased interest in ballroom dancing among participants, especially those with no prior experience. Many non-dancers reported enjoying the experience and expressed interest in seeing a more polished version of the app. This suggests the application could serve as an accessible introduction to ballroom dancing, providing users with an opportunity to try it out before committing to in-person group lessons. Additionally, it offers a foundational understanding of dance, potentially making the first formal lesson less intimidating. Although this wasn’t the app’s original goal, it represents an exciting prospect for future exploration.
Educational Value
Participants who completed the training experience reported that they had genuinely learned something about ballroom dancing. This outcome aligns with the app’s original purpose of being educational. Despite the issues identified, the app has already demonstrated its value as a learning tool. Addressing these issues would likely enhance its effectiveness, further solidifying its role as a training companion for learning ballroom dance.
Conclusion
In conclusion, I am proud of how the application came together, despite the significant hurdles and issues identified during the development process. This project provided me with a deeper understanding of the challenges and opportunities in creating a training tool for partner dancing in VR.
Looking forward, I would like to explore the potential of incorporating a multiplayer feature to enhance the experience. This could allow users to dance with remote partners or receive instruction from remote teachers, adding a new layer of connectivity and engagement.
As it stands, however, I believe this project has reached a stable and polished state, serving as a finished prototype that successfully explores the core concepts and viability of teaching ballroom dance in virtual reality.
Comments