CS 4616 Pattern Recognition

Spring 2007
CCB 102
MWF 1:00 - 2:00

Problem Sets        Syllabus        Projects

This course will cover basic and advanced techniques in Pattern Recognition. The goal of this class is to gain theoretical and practical understanding of classification techniques and their application to real-world problems.

Instructor

Jim Rehg
Email: rehg@cc.gatech.edu
Office: TSRB 230B
Office hours: After class or by appointment
Phone: 404-894-9105 (email preferred)

Prerequisites

Familiarity with basic probability and linear algebra. This is an introductory course and the lectures will be self-contained..

Text

The class textbook is Pattern Recognition and Machine Learning by Christopher M. Bishop. Springer 2006. There is a website with supplementary material.

Other texts:

• The Elements of Statistical Learning by T. Hastie, R. Tibshirani, and J. Friedman, Springer, 2001.
  Broad treatment of much of our course material from a statistician's perspective
• All of Statistics by L. Wasserman, Springer, 2004.
  Clear and concise summary of relevant probability and statistics material.
• Data Mining (2nd Ed.) by I. Witten and E. Frank, Morgan Kaufmann, 2005.
  Companion text to the Weka software (see below)
• Pattern Classification by Duda, Hart, and Stork. Wiley 2001.
  Updated version of the classic pattern recognition text
• Neural Networks for Pattern Recognition by C. Bishop. Oxford University Press, 1995. [Amazon]
• Linear Algebra and Its Applications or Introduction to Linear Algebra by G. Strang. [Amazon]
• Matrix Reference Manual [online]
• Introduction to Probability by D. P. Bertsekas and J. N. Tsitsiklis. [online]
• Probability, Random Variables, and Stochastic Processes by A. Papoulis.
  Classic text for probability theory and its application. Also see on-line lecture notes for EE178 at Stanford.
• Probability Theory: The Logic of Science by E. T. Jaynes. [online]
  Classic text on probability theory, chapters 1 and 2 in particular are good background reading.

Software

This semester we will be using the Weka software for machine learning and data mining for some of the assignments. Please download it and try it out, I have been using release 3-4-10. There is a sourceforge site with the complete code and documentation. The book by Witten and Frank (above) is a companion text to the software, but owning the book is not a requirement for using the package. In fact you may find that the online documentation is more helpful with respect to using the software.

Organization

Grades will be assessed as follows:

Collaboration on problem sets is encouraged at the "white board interaction" level. That is, share ideas and technical conversation, but write your own code, do your own detailed derivations, etc. A few problem sets may require you to work in teams of 2-3. I plan to grade and return problem sets promptly. As a result, I will require all problem sets to be turned in on time.

No late submissions will be accepted without prior permission of the instructor. If you need an extension, let me know in advance.

Problem Sets

• PS1: Handed out in class Mon Jan 29. Due in class Wed Feb 7. Data

Topics

• Introduction to Statistical Pattern Recognition
  • Review of multivariate probability and statistics
  • Example applications
• Bayesian Decision Theory
  • Philosophy and Methodology
  • Classifiers, Discriminant Functions, and Decision Surfaces
  • Neyman-Pearson Criteria and Bayes Risk
  • Discriminant Functions for Multivariate Normal Densities
  • Bayes Error and Loss Bounds
  • Application: Color-Based Skin Detection in Images
• Parameter Estimation
  • Maximum Likelihood Principle and Bayesian Estimation
  • Gaussian Learning in the Univariate and Multivariate Cases
  • Sufficient Statistics and the Exponential Family
  • Bias-Variance Dilemma
  • Computational Complexity
  • Application: Gene Expression Levels
• Nonparametric Methods
  • Kernel Density Estimators, Convergence Rates, and Error Bounds
  • Nearest Neighbor Rule, Convergence Rates, and Error Bounds
  • Computational Complexity and Dimensionality
  • Application: Robot Juggling Using Locally-Weighted Regression
• Feature Selection and Generation
  • Principal Components Analysis
  • Fisher’s Linear Discriminant
  • CART Trees
  • Multilayer Neural Networks
  • Application: Handwritten Character Recognition
• Model Selection
  • No Free Lunch Theorem
  • Occam’s Razor and Minimum Description Length Principle
  • Ensemble Methods: Bagging and Boosting
  • Bayesian Model Selection
  • Application: Audio Classification
    
• Learning Theory
  • VC Dimension
  • Structured Risk Minimization
  • Support Vector Machines
  • PAC Learning

Final Projects

I am very happy for you to do a final project that is related to some research project that you are involved in. I am also happy to work with to identify an appropriate final project topic.