In this paper we present a new approach for generating controlled animations
of video sprites. Video sprites are animations created by rearranging recorded
video frames of a moving object. With our technique, the user can specify
animations using a flexible cost function, which is automatically optimized by
repeated replacement of video sprite subsequences. We also show a fast technique
to compute video sprite transitions and a simple algorithm to correct for
perspective effects in the input footage. We use our techniques to create
character animations of animals, which are difficult both to train in the real
world and to animate as a 3D model.
The hamster data set was generated from 30 minutes of
hamster footage, yielding 15,000 usable sprite frames, or 30,000 after
mirroring. The first example animation uses a path constraint that makes the
hamster run around in a circle. The video on the website shows various
stages of the optimization, in increments of 30 seconds of processing time. |
|
The second example shows the hamster loitering at the
screen center, but he manages to move out of the way in time before a large
pile of conference proceedings crashes down. The optimization by subsequence
replacement can handle constraints that require a larger look-ahead than
beam search would be able to handle. To start, we use a weak location
constraint to keep the hamster near the center of the screen. At 4 seconds
later -- the time of impact -- we use a stronger hamster location constraint
that favors the far right corner, without specifying the specific path he
should take. The optimization correctly anticipates the change in cost and
moves the hamster out of harm's way. |
|
In the next scene we add another hamster that is moving to
the left side, crossing the first hamster's path. We use similar location
constraints for both hamsters, but add an anti-collision constraint between
the two. We optimize the sum of both hamsters' cost in turns of
approximately 100 random changes, always changing only one path while
keeping the other constant. |
|
The optimization can also incorporate character actions.
The next sequence shows the hamster entering the scene, and after following
a curved path that we created using the path constraint, he stands up on two
legs and looks at the audience. We manually classified all desirable input
sprite frames where the hamster is standing up facing the viewer, and once
the hamster reaches the bottom of the path, we add a frame constraint to the
cost function for 2 seconds. |
|
In the last hamster sequence, we add a second hamster that
is constrained to stay in a formation with the first using a location
constraint. The target location changes for each frame depending on the
first hamster's location and velocity. Additionally, the second hamster also
shares the first one's frame constraint, which makes the second stand up at
the same time as the first. |
|
For the second subject, the fly, we use one hour of raw
video footage, from which we extract, after automatically discarding
extended periods with no motion, 28,000 usable sprite frames, doubled to
56,000 by mirroring. A special challenge was that despite our efforts to
eliminate ranges of frames with no motion, most sprite images still showed a
fly moving in place instead of crawling. To generate the animation, we need
a more animated fly, which we create by modifying the approximating cost
function to favor
frames with large velocity. |
 |