Assignments:

1. Supervised Learning

2. Randomized Optimization

3. Unsupervised Learning and Dimensionality Reduction

4. Markov Decision Processes

Please note that all assignments are submitted via tsquare

An example of decent analysis

Implementation resources:

For the most part you should be able to find what you need for the assignments by searching the web, but in case you need it, here are some starting points.

• Kevin Murphy's MDP Toolbox for Matlab
• INRA's MDP Toolbox for Matlab
• Some MDP code in Java from the University of Rochester. I haven't looked at it in detail, so your mileage may vary...it seems to have policy and value iteration, though.
• Reinforcement Learning Repository at UMass, Amherst. Mostly has links to other stuff, including a C++ RL toolbox.
• A list of RL software at Rich Sutton's homepage. Again, mostly links to other stuff.
• Reinforcement Learning and Artificial Intelligence (RLAI). Has links to Python code (?).
• Unfortunately, I don't think there is a "standard" RL package out there (like Weka, for instance)...basically, just google for any combination of {MDP,Markov Decision Process[es],TD-lambda,reinforcement learning} + {code,software,package} [ + {Java, Matlab, C++, Python, ...} ].
• Frank Dellaert's Matlab Clustering Package.
• FastICA for Matlab courtesy of the Helsinki University of Technology. Also has FastICA for R, C++ and Python.
• Matlab already has PCA (princomp) and randomized projections are easy to implement.
• ABAGAIL, by Andrew Guillory. A java library of all the algorithms used by Andrew when he took 3600 and 4641.
• Weka 3: Data Mining Software in Java. Has classifiers and clustering algorithms.

Supplemental Reading:

1. Support Vector Machines and Kernel Methods (all taken from www.kernel-machines.org's tutorial page )

   1. Burges' tutorial on Support Vector Machines (risk, and VC dimension) (ps)
   2. Burbidge & Buxton's Introduction to SVMs (pdf). Some consider this a gentler introduction to SVMs.
   3. Scholkopf's NIPS tutorial slides on SVMs and kernel methods ps)

2. Boosting

   1. Schapire's Introduction to Boosting (ps)
   2. Boosting and margins (pdf)

3. Information Theory (pdf)

4. Randomized Search/MIMIC (pdf)

5. Ho and Pepyne's explanation of the No Free Lunch Theorem (.)

6. Berkhin's clustering survey (pdf)

7. Kleinberg's NIPS 2002 impossibility result for clustering (pdf)

8. Fodor's survey of dimensionality reduction techniques (pdf)

9. Rabiner's tutorial on Hidden Markov Models ( pdf) [you probably just want to read up to page 266 or so]

10. Reinforcement Learning
    1. Kaelbling, Littman and Moore's survey of reinforcement learning (pdf)
    2. Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto. Check out the link to the HTML version at the bottom of the intro page.
    3. Michael Littman's slides on RL. In particular, the last slide contains the simplified TD-lambda update rule we covered in class.

11. Andrew Moore's tutorial slides on zero-sum game theory and non-zero-sum game theory.

12. Shoham, Powers and Grenager's survey on multi-agent learning

13. Model selection and overfitting (pdf)