I am making a game in Unity that involves creatures whose animations are determined by physics. For example, to move a limb of a creature, I can apply forces to the rigidbody it's associated with. I know how to apply forces programmatically through a script to create movements, but I'd like to create more complex and organic movements and thought that I might be able to use a neural network to do this.
I'd like each of the creatures to have a distinct way of moving in the world. I'd like to first puppeteer the creatures manually using my hand (with a Leap Motion controller), and have a neural network generate new movements based on the training I did with my hand.
More concretely, my manual puppeteering setup will apply forces to the rigidbodies of the creature as I move my hand. So if I lift my finger up, the system would apply a series of upward forces to the limb that is mapped to my finger. As I am puppeteering the creature, the NN receives Vector3 forces for each of the rigidbodies. In a way this is the same task as generating a new text based on a corpus of texts, but in this case my input is forces rather than strings.
Based on that training set, is it possible for the NN to generate movements for the characters (forces to be applied to the limbs) to mimic the movements I did with my hand?
I don't have that much experience with neural networks, but am eager to learn, specifically for this project. It would be great to know about similar projects that were done in Unity, or relevant libraries I could use that would simplify the implementation. Also, please let me know if there is anything I can clarify!
Not really an answer but would not fit for comments
I'm not sure the strategy you want to apply to train your model is the right one.
I would go for reinforcement learning methods (you can check this question for more infos about it) using, for example, the distance traveled by the center of mass of the creature on the x-axis as a fitness. If this leads to weird behaviours (like this well known robot) you could, for example, think of strategies like penalizing your individuals given the distance traveled on y and z axis (still by the CoM) to try having guys that keep there CoM on the same plane.
Without knowing exactly what you want to achieve this is hard to give you more advices. Although, if you are not looking only for neural network based techniques, there is this really great paper you might want to have a look at (here is the video of their results).