VR Person Project

This project is an idea I had for the 100 Days of Code Challenge. My list of ideas essentially stated:

Make a Unity Pet/Person/Slime Blob that I can interact with in VR

• 3D Pose from Video
• Audio Clone of Voice
• Can it be a person’s face and figure from a video then a 3d model of them doing it in VR (like watching kpop dance videos)

Which is a pretty cute idea. To elaborate more, I think what I want is to train a model to take in a video as input and output an animated 3D model that can be displayed in VR.

The model should move the way the person is moving in the video which would involve 3D Pose Estimation.

Project Specifications

The basic skeleton:

• Input: Video of a person moving
• Output: 3D Animated Model moving the same way

Additional Features (good to have in the future):

• Implement a speech feature that sounds like that person
  • Relevant PaperGithub
• Can apply a face onto the model
• Model for non-human characters (like a cat)
• Change the movement of the characters (so you rig the character differently) but with the same model (it looks like the person in the video)
• Compare character models for 2 videos and highlight parts which differ (useful for learning proper form)

What this could be used for

Honestly, I’m doing this project because I think it will be cool, fun and I’ll learn a lot about 3D Human Pose Estimation (and related topics) but I’m sure there are some usages for this:

• Learn how to do some actions properly
  • Weight-lifting forms (which areas to pay attention to)
  • K-Pop Dance Moves
• Have fun watching people in 3D
  • Concerts (It’ll be like you were actually there)

Research

For each topic covered, I will have some mini-projects within the blog post related to it to familiarise myself with the topic/library.

Main Topics

• 3D Human Pose Estimation
• Character Animation in VR

Potential Areas to look into?:

• Generating Video based on Character Rigging (for rig the character differently so it looks like the person in the video is moving differently) – I know this has been done with speech & videos of the face.
• Apply a face onto the model – Should be similar to DeepFakes

<TODO> About 3D Human Pose Estimation

<TODO> Will link summaries of relevant papers soon

Progress

<TODO> Will update and link a github