README

AI4Animation: Deep Learning for Character Control
============

This repository explores the opportunities of deep learning for character animation and control. It aims to be a comprehensive framework for data-driven character animation, including data processing, neural network training and runtime control. The various projects below demonstrate such capabilities using neural networks for animating biped locomotion, quadruped locomotion, and character-scene interactions with objects and the environment, plus sports and fighting games, as well as embodied avatar motions in AR/VR. Further advances on this research will continue being added to this project.

------------
2026 - Python Remake (AI4AnimationPy)
View repository  →

Paul Starke, 
Sebastian Starke

------------

This version brings AI4Animation to Python — removing the Unity dependency for data-processing, feature-extraction, inference, and post-processing while keeping similar game-engine-style architecture (ECS, update loops, rendering pipeline). Everything runs on NumPy or PyTorch, so training, inference, and visualization happen in one unified environment which helps streamlining the research process.



Below are some examples generated in AI4AnimationPy which can be directly run from the demos folder and supposed to give an idea what can be done with the framework:

| | |
|---|---|
| !Locomotion Demo | !Quadruped Demo |
| Stylized Biped Locomotion Controller trained on style100 | Quadruped Locomotion Controller — Interactive dog locomotion with gait transitions and action poses |
| !Training Demo | !ECS Demo |
| Future Motion Anticipation with Interactive model training visualization | ECS — Entity hierarchy and component system |
| !IK Demo | !MocapImport Demo |
| Inverse Kinematics — Real-time IK solving | Motion Capture Import — GLB/FBX/BVH/NPZ loading |
| !MotionEditor Demo | |
| Motion Editor — animation browsing and feature visualization| |

Play Web Demos  


    
    
    
    
    
    


------------
SIGGRAPH 2024
Categorical Codebook Matching for Embodied Character Controllers

Sebastian Starke, 
Paul Starke, 
Nicky He, 
Taku Komura,
Yuting Ye, 
ACM Trans. Graph. 43, 4, Article 142.

------------




Translating motions from a real user onto a virtual embodied avatar is a key challenge for character animation in the metaverse. In this work, we present a novel generative framework that enables mapping from a set of sparse sensor signals to a full body avatar motion in real-time while faithfully preserving the motion context of the user. In contrast to existing techniques that require training a motion prior and its mapping from control to motion separately, our framework is able to learn the motion manifold as well as how to sample from it at the same time in an end-to-end manner. To achieve that, we introduce a technique called codebook matching which matches the probability distribution between two categorical codebooks for the inputs and outputs for synthesizing the ch