Controlled Animation of Video Sprites



Paper (10.2 MB .pdf)

In this paper we present a new approach for generating controlled animations of video sprites. Video sprites are animations created by rearranging recorded video frames of a moving object. With our technique, the user can specify animations using a flexible cost function, which is automatically optimized by repeated replacement of video sprite subsequences. We also show a fast technique to compute video sprite transitions and a simple algorithm to correct for perspective effects in the input footage. We use our techniques to create character animations of animals, which are difficult both to train in the real world and to animate as a 3D model.

Slides for Panel on Animation at SIGGRAPH 2002


The hamster data set was generated from 30 minutes of hamster footage, yielding 15,000 usable sprite frames, or 30,000 after mirroring. The first example animation uses a path constraint that makes the hamster run around in a circle. The video on the website shows various stages of the optimization, in increments of 30 seconds of processing time.


The second example shows the hamster loitering at the screen center, but he manages to move out of the way in time before a large pile of conference proceedings crashes down. The optimization by subsequence replacement can handle constraints that require a larger look-ahead than beam search would be able to handle. To start, we use a weak location constraint to keep the hamster near the center of the screen. At 4 seconds later -- the time of impact -- we use a stronger hamster location constraint that favors the far right corner, without specifying the specific path he should take. The optimization correctly anticipates the change in cost and moves the hamster out of harm's way.


In the next scene we add another hamster that is moving to the left side, crossing the first hamster's path. We use similar location constraints for both hamsters, but add an anti-collision constraint between the two. We optimize the sum of both hamsters' cost in turns of approximately 100 random changes, always changing only one path while keeping the other constant.


The optimization can also incorporate character actions. The next sequence shows the hamster entering the scene, and after following a curved path that we created using the path constraint, he stands up on two legs and looks at the audience. We manually classified all desirable input sprite frames where the hamster is standing up facing the viewer, and once the hamster reaches the bottom of the path, we add a frame constraint to the cost function for 2 seconds.


In the last hamster sequence, we add a second hamster that is constrained to stay in a formation with the first using a location constraint. The target location changes for each frame depending on the first hamster's location and velocity. Additionally, the second hamster also shares the first one's frame constraint, which makes the second stand up at the same time as the first.


For the second subject, the fly, we use one hour of raw video footage, from which we extract, after automatically discarding extended periods with no motion, 28,000 usable sprite frames, doubled to 56,000 by mirroring. A special challenge was that despite our efforts to eliminate ranges of frames with no motion, most sprite images still showed a fly moving in place instead of crawling. To generate the animation, we need a more animated fly, which we create by modifying the approximating cost function to favor frames with large velocity.