There are existing github libraries that already implement the 3D Human Pose Estimation I need: I’m mainly looking at DensePose Github, V-Nect Github and VideoPose3D.
Can it be a person’s face and figure from a video then a 3d model of them doing it in VR (like watching kpop dance videos)
Which is a pretty cute idea. To elaborate more, I think what I want is to train a model to take in a video as input and output an animated 3D modelthat can be displayed in VR.
The model should move the way the person is moving in the video which would involve 3D Pose Estimation.
Project Specifications
The basic skeleton:
Input: Video of a person moving
Output: 3D Animated Model moving the same way
Additional Features (good to have in the future):
Implement a speech feature that sounds like that person
Change the movement of the characters (so you rig the character differently) but with the same model (it looks like the person in the video)
Compare character models for 2 videos and highlight parts which differ (useful for learning proper form)
What this could be used for
Honestly, I’m doing this project because I think it will be cool, fun and I’ll learn a lot about 3D Human Pose Estimation (and related topics) but I’m sure there are some usages for this:
Learn how to do some actions properly
Weight-lifting forms (which areas to pay attention to)
K-Pop Dance Moves
Have fun watching people in 3D
Concerts (It’ll be like you were actually there)
Research
For each topic covered, I will have some mini-projects within the blog post related to it to familiarise myself with the topic/library.
Main Topics
3D Human Pose Estimation
Character Animation in VR
Potential Areas to look into?:
Generating Video based on Character Rigging (for rig the character differently so it looks like the person in the video is moving differently) – I know this has been done with speech & videos of the face.
Apply a face onto the model – Should be similar to DeepFakes
<TODO> About 3D Human Pose Estimation
<TODO> Will link summaries of relevant papers soon
I’ll be going to a hackathon later this month and decided to dedicate some time to figure out what kind of hacks are most likely to win.
About Hackathons in General
Generally you should present something that has some practical purpose at the end of the hackathon, or a prototype of such a thing.
It would be a bonus if the project is using cutting-edge technology / does something that seems really hard to do.
You cannot code before the hackathon.
About JunctionX
I’ll be attending JunctionX with my best friend 😀 to learn some new stuff, have some fun and hopefully win a prize!
Credit: JunctionX Singapore
Junction X defines a hackathon as:
A hackathon is an event in which small teams (2-5 person) of developers, designers, entrepreneurs and other specialists collaborate intensively on software projects aimed at solving particular problems. Within 48 hours, teams should come up with a working prototype and a presentation.
First you will be judged by other participants, then if you are the top 3, you will be judged by the partners.
Partners will be setting the criteria… so probably something useful from a corporate sense?
Presentation will be 3 minute demo and 2 minute Q&A – so present fast, clearly with a main purpose of your product, probably how it fits the criteria then make sure you know your creation well 😀
As mentioned before, no code written before event. You can however use open source libraries so I can look up relevant libraries for my ideas and figure out how to do stuff e.g. how to code for transfer learning if its a language/library I’m unfamiliar with.
Past Junction Hackathons
We can look at the general type of project:
Cutting Edge Tech / Relative New Tech
Social Good / Corporate Useful Thing
Cute Useful Thing
We can also look at the judges comments:
Glados’ hack Signvision was praised by the main judges for it’s awesome technical implementation (said by Jari Jaanto who read through all the finalists’ code) and impressive user experience. The hack was said have real potential and that it was practically ready to be deployed.
Junction 2017 Winner: A Mix of Machine Learning and Cool UX
Usually the judges will tell us their criteria on the day of the hackathon along with the themes so I should just come up with a few ideas that can be implemented. I’m pretty much going to work along my current interests because even if I don’t win, I do want to have fun.
So one idea could be to
use computer vision to get input from some videos (filmed on the day itself) and use that to generate a model (using transfer learning from existing state of the art models) and make something cute in VR.
Misc
This is what the schedule looks like, I’ll probably head home to sleep since it is a pretty long hackathon and past a certain level of sleep deprivation, I can’t code effectively.
Increase my knowledge on my field of research for my PhD which will be starting in January
Familiarise myself with a wider range of code libraries (hopefully the next time I go to a hackathon, I’ll make something really cool!)
Be a more productive coder by customising my workflow and making useful little tools
Make something that displays my skills for my portfolio
Today, I’m going to plan what to do during this challenge. Since doing various projects will generally lead me towards new ideas, I’m going to just have a list of projects I can work on.
About Projects
I’m going to list some project ideas for each of my goals. I’ll come back and refine the list of projects when I have more ideas.
PhD Research
I will generally be reading some papers then trying to implement that in some little project to try out what I’ve read so I can understand what that paper is doing. Implementation usually takes quite a bit of time, especially for the projects which require training an AI (I have a macbook… and a VR gaming computer ;D ) unless the paper has a really good github repo.
I am interested in AI: Reinforcement Learning (Communication between multiple agents), Computer Vision (3D Pose Estimation) and Human Computer/Robot interaction.
Hackathon – Make Something Cool
I usually use Unity to create a cool game in Hackathons (like VR games) or rig up something with an Arduino and Raspberry Pi. I really like projects that are useful, fun and use technology to do unexpected things.
Interesting to Learn
These would be fields that I’ve always wanted to explore or cute thing I wanted to create to make my life easier.
Data science visualisation
Libraries to make tutorial videos
Manim (Here is my tutorial on installing it) – I haven’t tried this out yet
VR Scene Design
Productive Coder
These projects will be small and quick, usually I’ll script up a tool to shorten my workflow. An example of projects like this would be Alfred scripts, trying out Emacs and learning how to use some tool (like Git) better.
Skills Portfolio
These projects would be aimed at showing my mastery of some skill, toolbox etc
Project Ideas
TODO: add time estimates to the projects so on busy days I can pick a shorter project to work on.
Research
Make a Unity Pet/Person/Slime Blob that I can interact with in VR
I found that MacTex didn’t really work for me and I didn’t need to use the last step in Brian Howell’s tutorial so this is an updated and succinct version of his post. His post does cover the first few steps in greater detail if needed
Step 1: Install packages
Paste the following lines into terminal one by one.
Create a folder where you want the Manim library to be stored in, I recommend you pick one that is only one word like “Animations”. I have a folder named “Projects” in my home directory. I created the Animations folder in the Projects folder so my path is essentially “~/Projects/Animations”
Open a new terminal and cd to that folder as shown below
cd <Path-to-Newly-Created-Directory>
Example of changing directory in terminal
Next we will download that manim37 folder in the image below (at this moment your animation directory should still be empty)
git clone https://github.com/3b1b/manim.git
Next, rename the manim folder that has just been downloaded to manim37 (you don’t have to do this but the following steps does use manim37 as the folder name)
Step 4: Virtual Environment Install Packages
The following commands will change your directory to the manim37 folder, setup and activate a virtual environment and do the package installations for dependencies.
In the manim37 folder, create a folder called “output”
Step 6: Modify File Path
Open the constants.py found in manim37/manimlib
Where the constants.py is located
Change line 27 (the line that says “Dropbox (3Blue1Brown)/3Blue1Brown Team Folder”) to the path to your output folder
My path to the output folder is “~/Projects/Animations/manim37/output” so I will change line 26 to “Projects/Animations/manim37/output”
What the constants.py should look like
Step 7: Test & Fixes
Basic Test
Do a test to check if the basic animations are working by entering the following into the terminal
python3 -m manim example_scenes.py
2
You want to select the SquareToCircle example scene so the response is 2 as shown above
What it should look like if you succeeded
You can play the SquareToCircle.mp4 to see if you get a white diamond outline turning into a purple filled red outlined circle
Latex Test for Text Animations
Test if text animations are working by entering the following into the terminal:
python3 -m manim example_scenes.py
5
You want to select the WriteStuff example scene so the response is 5 as shown above.
What it should look like if you succeeded
If this part fails, it is likely because there is something wrong with your latex. I recommend going to the .log files inside the output/Tex folder
Where the log files are located
You might find lines like:
!LaTeX Error: File `ragged2e.sty' not found.
You can go to MikTex Console (the app you downloaded previously) and search for it in the packages list. To download it, select the package and click the + icon.
The MikTex Console should prompt you when you are running the test program to ask if you want to download the packages needed.
caffae 