Propagation Networks for Recognizing Partially Ordered Sequential Activity

Yifan Shi, Yan Huang, David Minnen, Aaron Bobick, Irfan Essa

 

This work is presented in CVPR'2004, (PDF, BIB)

Goals

Represent and fuse human knowledge of daily activities with noisy perceptual features
Detect and recognize an activity
Pinpoint components of the activity and detect missing or improperly performed steps

Limitations of Existing Approaches

Difficulty in modeling parallel streams

Finite State Automata

Bayesian Network

Hidden Markov Model

Dynamic Bayesian Network

Stochastic Context-free Grammar

Difficulty in handling noisy sensor data

PNF-Network

Contributions

Handle multiple, parallel streams
Naturally represent partially ordered activity components
Directly model duration of components
Deal with noisy underlying vision sensor data
Classify whole sequence as well as identify individual components
Applied to recognition of actions in the medical domain

Propagation Network (P-Net)

Use links to represent temporal partial ordering;
Use nodes with duration model to represent temporal interval;
Represent parallel activity stream with multiple, simultaneously active nodes;
Use conditional probability function to represent temporal/logical relationship between nodes;
Embed duration model in activation triggering function;
Use hidden observation model for underlying vision uncertainty

Recognizing with P-Net

Local Maximal Search Algorithm 
Viterbi-like forward searching
Backward check for path consistency
D-Condensation
Discrete-state particles used for efficient inference in P-Net;
Each particle represents the state of all nodes representing one coherent explanation of the observation;
Generate all possible offspring of each particle during update;
Best N particles survive, allowing scalable performance;
Normalize particle probabilities during state evolution avoiding redundant calculations increasing computational efficiency

Experiment: Blood Glucose Monitor Calibration

Model the complex LifeScan glucose calibration process with a P-Net using simple vision detectors
3 subjects with 41 videos
3 task categories:
Correct: all nodes present. P-Net identifies them all
Almost right: one node missing. P-Net incorrectly accepts 20% and correctly reports missing steps in the other 80%.
Negative: missing many steps. P-Net correctly rejects half of the test sequences, others are accepted, but P-Net correctly identifies the missing steps.

P-Net Definition for Glucose Meter Calibration Task

 

System Architecture

Detection rate for individual nodes over all samples

Ability to locate critical moment

 

TurnOn [44]

 Unscrew Cap [229] Drop Solution [256] Screw Cap [288] Insert Strip [329] Mark Ctrl [655]
Snapshots from example sequence and corresponding detection results

Full video clips

Dataset:

Subject1 correct sequence 1 subject1_right1.avi
Subject1 correct sequence 2 subject1_right2.avi
Subject1 correct sequence 3 subject1_right3.avi
Subject1 correct sequence 4 subject1_right4.avi
Subject1 correct sequence 5 subject1_right5.avi
Subject1 correct sequence 6 subject1_right6.avi
Subject1 correct sequence 7 subject1_right7.avi
Subject1 correct sequence 8 subject1_right8.avi
Subject1 correct sequence 9 subject1_right9.avi
Subject1 correct sequence 10 subject1_right10.avi
Subject2 correct sequence 1 subject2_right1.avi
Subject2 correct sequence 2 subject2_right2.avi
Subject2 correct sequence 3 subject2_right3.avi
Subject2 correct sequence 4 subject2_right4.avi
Subject2 correct sequence 5 subject2_right5.avi
Subject2 correct sequence 6 subject2_right6.avi
Subject2 correct sequence 7 subject2_right7.avi
Subject2 correct sequence 8 subject2_right8.avi
Subject2 correct sequence 9 subject2_right9.avi
Subject2 missing-one-step sequence 1 subject2_noshake1.avi
Subject2 missing-one-step sequence 2 subject2_noshake2.avi
Subject2 missing-one-step sequence 3 subject2_noshake3.avi
Subject2 missing-one-step sequence 4 subject2_noshake4.avi
Subject2 missing-one-step sequence 5 subject2_noshake5.avi
Subject2 missing-one-step sequence 6 subject2_noshake6.avi
Subject2 missing-one-step sequence 7 subject2_noshake7.avi
Subject2 missing-six-step sequence 1 subject2_noapply1.avi
Subject2 missing-six-step sequence 2 subject2_noapply2.avi
Subject2 missing-six-step sequence 3 subject2_noapply3.avi
Subject2 missing-six-step sequence 4 subject2_noapply4.avi
Subject2 missing-six-step sequence 5 subject2_noapply5.avi
Subject3 correct sequence 1 subject3_right1.avi
Subject3 correct sequence 2 subject3_right2.avi
Subject3 missing-one-step sequence 1 subject3_noshake1.avi
Subject3 missing-one-step sequence 2 subject3_noshake2.avi
Subject3 missing-one-step sequence 3 subject3_noshake3.avi
Subject3 missing-six-step sequence 1 subject3_noapply1.avi
Subject3 missing-six-step sequence 2 subject3_noapply2.avi
Subject3 missing-six-step sequence 3 subject3_noapply3.avi
Subject3 missing-six-step sequence 4 subject3_noapply4.avi
Subject3 missing-six-step sequence 5 subject3_noapply5.avi

 

 

 

 

Home

Projects

Publications

People

Courses

Sponsors

Co-Web
CPL Swiki

GVU

Copyright © 1997-2005

Last Updated Dec 31, 2004.