CS 7643 Deep Learning

Fall 2017, TR 4:30 - 5:45 pm, Clough 144

Course Information

This is an exciting time to be studying (Deep) Machine Learning, or Representation Learning, or for lack of a better term, simply Deep Learning!

Deep Learning is rapidly emerging as one of the most successful and widely applicable set of techniques across a range of domains (vision, language, speech, reasoning, robotics, AI in general), leading to some pretty significant commercial success and exciting new directions that may previously have seemed out of reach.

This course will expose students to cutting-edge research — starting from a refresher in basics of neural networks, to recent developments. The course is structured in “modules” (background, CNNs, RNNs, Deep Structured Prediction, Deep Reinforcement Learning). Each “module” will begin with instructor lectures to present context and background material. The emphasis will then switch to student-led paper presentations and discussions.

Instructor: Dhruv Batra
Teaching Assistants: Abhishek Das
Michael Cogswell
Zhaoyang Lv
Office Hours: Abhishek Das: Thursday, 11:00 am - 12:00 pm, CCB 2nd Floor Common Area
Michael Cogswell: Wednesday, 6:00 pm - 7:00 pm, CCB 2nd Floor Common Area
Zhaoyang Lv: Tuesday, 3:30 pm - 4:30 pm, CCB 2nd Floor Common Area
Class meets: Tue, Thu 4:30 - 5:45 pm, Clough 144
Piazza: piazza.com/gatech/fall2017/cs7643
Canvas: gatech.instructure.com/courses/772
Staff Mailing List: cs-7643-f17-staff@googlegroups.com

Schedule

PS = Problem Set (written)
HW = Homework (implementation)
HR = Historically Relevant

Date	Topic	Primary Reading	Optional Reading
W1: Aug 22	Class Administrativia, HW0 out Slides (pptx), Slides (pdf).		LeCun et al., Nature '15 Shannon, 1956
W1: Aug 24	HW0 due No class
W2: Aug 29	Background: Notation + supervised learning review, linear classifiers, modular design of neural networks Slides, Slides (annotated).
W2: Aug 31	Background: Regularization, modular design of neural networks, computing gradients Slides, Slides (annotated), Notes.
W2: Sep 1	PS1 out
W3: Sep 5	Computational Graphs, Computing Gradients, Forward mode vs Reverse mode AD Slides, Slides (annotated),		Automatic Differentiation Survey, Baydin et al.
W3: Sep 6 12:30 - 1:30 pm (Lunch at Noon) Marcus Nanotechnology Building 1118	Invited talk: Soumith Chintala PyTorch, Static vs. Dynamic Computation Graphs
W3: Sep 7	Invited talk: Nathan Silberman TF-Slim: A lightweight library for defining, training and evaluating complex models in TensorFlow
W3: Sep 8	PS1 due HW1 out
W4: Sep 12	No class
W4: Sep 14	(Finish) Forward mode vs Reverse mode AD, and Convolutions. Slides, Slides (annotated), Notes.
W5: Sep 19	CNNs: Convolutions, stride, padding, 1x1 convolutions, Slides, Slides (annotated), Notes.
W5: Sep 21	transposed convolutions, dilated/atrous convolutions; CNN Architectures: AlexNet, NIN, VGG/GoogLeNet, ResNets Slides, Slides (annotated).
W5: Sep 22	HW1 due
W6: Sep 26	Segmentation CNNs (and other pixel-level prediction) Invited talk: Sumit Chopra PS2 out (tex source) HW2 out	Fully Convolutional Networks for Semantic Segmentation	U-Net: Convolutional Networks for Biomedical Image Segmentation DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation Pyramid Scene Parsing Network FlowNet: Learning Optical Flow with Convolutional Networks Depth Map Prediction from a Single Image using a Multi-Scale Deep Network Colorful Image Colorization
W6: Sep 28	Visualizing CNNs Slides, Slides (annotated).	Methods for Interpreting and Understanding Deep Neural Networks	Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps Visualizing and Understanding Convolutional Networks Visualizing Deep Neural Network Decisions: Prediction Difference Analysis Understanding Deep Image Representations by Inverting Them Learning Deep Features for Discriminative Localization Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization Network Dissection: Quantifying Interpretability of Deep Visual Representations
W7: Oct 3	No class
W7: Oct 5	Invited talk: Stefan Lee An Introduction to Visually Grounded Question Answering, Dialog, and Beyond
W8: Oct 10	Fall student recess: no class
W8: Oct 12	Fooling CNNs and Adversarial Examples	Explaining and Harnessing Adversarial Examples	Intriguing properties of neural networks DeepFool: a simple and accurate method to fool deep neural networks Foveation-based Mechanisms Alleviate Adversarial Examples Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks Towards Evaluating the Robustness of Neural Networks On Detecting Adversarial Perturbations Universal adversarial perturbations Practical Black-Box Attacks against Machine Learning Adversarial Examples for Semantic Segmentation and Object Detection SafetyNet: Detecting and Rejecting Adversarial Examples Robustly Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
W8: Oct 13	PS2 due, HW2 due
W8: Oct 14	PS3 out (tex source) HW3 out
W9: Oct 17	Recurrent Neural Networks Slides, Slides (annotated), Notes.		A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation by Gang Chen.
W9: Oct 19		Sequence to Sequence Learning with Neural Networks	Neural Machine Translation by Jointly Learning to Align and Translate Visualizing and Understanding Recurrent Networks Visualizing and Understanding Neural Models in NLP Pixel Recurrent Neural Networks Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues Batch Normalized Recurrent Neural Networks Recurrent Batch Normalization Very Deep Convolutional Networks for Text Classification Neural Machine Translation in Linear Time
W10: Oct 24	ICCV: no class
W10: Oct 26	ICCV: no class
W10: Oct 28	PS3 due, HW3 due
W11: Oct 31	CNNs + RNNs	Show and Tell: A Neural Image Caption Generator	Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning Context-aware Captions from Context-agnostic Supervision VQA: Visual Question Answering Stacked Attention Networks for Image Question Answering CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning A simple neural network module for relational reasoning Visual Dialog
W11: Nov 2	Deep Reinforcement Learning Slides, Slides (annotated).
W12: Nov 7		Sequence Level Training with Recurrent Neural Networks	Self-critical Sequence Training for Image Captioning Improved Image Captioning via Policy Gradient optimization of SPIDEr Learning to Reason: End-to-End Module Networks for Visual Question Answering Inferring and Executing Programs for Visual Reasoning Modular Multitask Reinforcement Learning with Policy Sketches Recurrent Models of Visual Attention Deal or No Deal? End-to-End Learning for Negotiation Dialogues Deep Reinforcement Learning for Dialogue Generation Towards Diverse and Natural Image Descriptions via a Conditional GAN REINFORCE (Williams92) Mastering the game of Go with deep neural networks and tree search Asynchronous Methods for Deep Reinforcement Learning
W12: Nov 9		Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning	End-to-end optimization of goal-driven and visually grounded dialogue systems Multi-Agent Cooperation and the Emergence of (Natural) Language Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols Translating Neuralese Emergence of Grounded Compositional Language in Multi-Agent Populations Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Analogs of Linguistic Structure in Deep Representations Emergent Language in a Multi-Modal, Multi-Step Referential Game
W13: Nov 14	CVPR Deadline: no class
W13: Nov 16		Mapping Instructions and Visual Observations to Actions with Reinforcement Learning	Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning Visual Semantic Planning using Deep Successor Representations Cognitive Mapping and Planning for Visual Navigation Gated-Attention Architectures for Task-Oriented Language Grounding Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning Grounded Language Learning in a Simulated 3D World Learning to Navigate in Complex Environments Playing Doom with SLAM-Augmented Deep Reinforcement Learning
W14: Nov 21	Variational Autoencoders and Generative Adversarial Networks Slides, Slides (annotated).
W14: Nov 23	Thanksgiving break: no class
W15: Nov 28		Tutorial on Variational Autoencoders	Auto-Encoding Variational Bayes Semi-supervised Learning with Deep Generative Models Categorical Reparameterization with Gumbel-Softmax Variational Dropout and the Local Reparameterization Trick beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework Improving Variational Inference with Inverse Autoregressive Flow
W15: Nov 30		Generative Adversarial Networks	Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Generative Adversarial Text to Image Synthesis LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation Wasserstein GAN Energy-based Generative Adversarial Network Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
W16: Dec 5	NIPS: no class

Grading

50% Homework (3 homeworks + 3 problem sets)
15% Paper Presentation
10% Paper Reviews
20% Final Project
5% Class Participation

Late policy for deliverables

No penalties for medical reasons (bring doctor’s note) or emergencies.
Every student has 7 free late days (7 x 24-hour chunks) for this course; these cannot be used for HW0, the reviews or the presentation.
After all free late days are used up, penalty is 25% for each additional late day.

Prerequisites

CS 7643 is an ADVANCED class. This should not be your first exposure to machine learning. Ideally, you need:

Intro-level Machine Learning
- CS 7641/ISYE 6740/CSE 6740 or equivalent
Algorithms
- Dynamic programming, basic data structures, complexity (NP-hardness)
Calculus and Linear Algebra
- positive semi-definiteness, multivariate derivates (be prepared for lots and lots of gradients!)
Programming
- This is a demanding class in terms of programming skills.
- HWs will involve a mix of languages (Python, C++) and libraries (PyTorch).
- Your language of choice for project.
Ability to deal with abstract mathematical concepts

FAQs

The class is full. Can I still get in?

It is unlikely. The class is full, with a 200 person wait-list. It is unlikely that the class size will increase further. We recommend coming to the first class. There will be a HW0 released on day 1 to check background preparation. Our past experience with HW0 suggests that slots will open up after some students drop the class.
I have not taken an graduate-level “Machine Learning” class or I am taking it in parallel. Can I still take this class?

No. Graduate-level machine learning is a pre-requisite.
Can I audit this class or take it pass/fail?

No. Due to the large demand for this class, we will not be allowing audits or pass/fail. Letter grades only. This is to make sure students who want to take the class for credit can.
Can I simply sit in the class (no credits)?

In general, we welcome members of the Georgia Tech community (students, staff, and/or faculty) to sit-in. Out of courtesy, we would appreciate if you let us know beforehand (via email or in person). If the classroom is full, we would ask that you please allow registered students to attend.
I have a question. What is the best way to reach the course staff?

Registered students – your first point of contact is Piazza (so that other students may benefit from your questions and our answers). If you have a personal matter, email us at the class mailing list cs-7643-f17-staff@googlegroups.com.

Projects

You are encouraged to try out interesting applications of deep learning (vision, NLP, computational biology, UAVs, etc!)
The project must be done in this semester. No double counting.
You may combine with other course project but must delineate the different parts
Extra credit for shooting a publication
Main Categories:
- Application/survey - compare a bunch of algorithms on a new application domain of your interest
- Formulation/Development - Formulate a new model or algorithm for a new/old problem
- Theory: Theoretically analyze an existing algorithm.

Book

Deep Learning, Ian Goodfellow, Aaron Courville, and Yoshua Bengio, MIT Press

Overviews

Note to people outside Georgia Tech

Feel free to use the slides and materials available online here. If you use our slides, an appropriate attribution is requested. Please email the instructor with any corrections or improvements.