VLG Group
Group Meetings
Y. LeCun's website
CS at Courant
Courant Institute

Machine Learning and Pattern Recognition: Schedule

[ Course Homepage | Schedule and Course Material | Mailing List ]

This page contains the schedule, slide from the lectures, lecture notes, reading lists, assigments, and web links.

I urge you to download the DjVu viewer and view the DjVu version of the documents below. They display faster, are higher quality, and have generally smaller file sizes than the PS and PDF.

Full-text search is provided for the entire collection of slides and papers. Click here to search

You can have a look at the schedule and class material for the graduate version of this course taught during the Fall 2004 semester, but be warned that the undergraduate edition is significantly different.

01/18: Introduction and basic concepts

Subjects treated: Intro, types of learning, nearest neighbor, how biology does it, linear classifier, perceptron learning procedure, linear regression,

Slides: [DjVu | PDF | PS]

01/20: Introduction to Lush, Computer Lab

IMPORTANT NOTE: This session will start in the usual classroom WWH-102, but the second part will be in room WWH-512 (Sun computer lab on the 5-th floor).

Subjects treated: An introduction to the Lush programming language. Implementing the perceptron learning algorithm.

The instructor for this lab seesion will be Fu-Jie Huang (Jie).

You should all have a login on the Sun machines in the lab.

If you have laptop, you are welcome to install Lush on it and to use it for the computer lab sessions instead of the labs's Sun workstations. Lush runs on Linux, Mac OS-X, Solaris, and on Windows under Cygwin.

Click here to get the files for the lab and the homework.

HOMEWORK 01b: The homework is due 02/08 before the class (NOTE: new due date).

Slides: [DjVu | PDF | PS]

Lush Tutorial: read the short tutorial on Lush at the Lush documentation page.

01/25: Perceptron, Linear Regression

Subjects treated: Intro, perceptron convergence theorem, multivariate calculus refresher, loss functions, linear regression, LMS/Adaline.

Slides: [DjVu | PDF | PS]

02/27: Energy-Based Models, Loss Functions, Linear Regression

Subjects treated: Energy-Based Models, Inference, Loss Functions, Logistic Regression.

Slides: [DjVu | PDF | PS]

02/01: Lab: Linear Classifiers

IMPORTANT NOTE: We will meet in the usual classroom WWH-102, and promptly go to the Sun computer lab in room WWH-512. .

Subjects treated: LAB: perceptron, LMS, Logistic Regression.

HOMEWORK 02a: Click here to get the files for the lab and the homework.

To uncompress the homework file on Unix/Linux do "tar xvfz lab-02a.tgz". This will create a directory named "lab-02a". On Windows, use Winzip.

The homework is due 02/15 before the class.

02/03: Generalization, Regularization

Subjects treated: Learning and Generalization, Regularization.

Slides: [DjVu | PDF | PS]

02/08: Multi-Module Systems, Backpropagation

Subjects treated: Multilayer and Multi-Module Systems, Gradient Back-Propagation.

Slides: [DjVu | PDF | PS]

02/10: Modules and Architectures

Subjects treated: Modules and Architectures.

Slides: [DjVu | PDF | PS]

02/15: Special Architectures

Subjects treated: Special Architectures for Time Series, Images, Audio, Video.

Slides: [DjVu | PDF | PS]

02/17: Audio and Image Classification

Subjects treated: Convolutional Networks for speech and image classification.

Slides: object recognition [DjVu].

HOMEWORK 05b: computing Jacobians. Click one these links to get the text of the homework: [DjVu | PDF | PS]

The homework is due 03/01 before the class.

Required Reading:

Gradient-based Learning Applied to Document Recognition by LeCun, Bottou, Bengio, and Haffner; pages 1 to the first column of page 18: [DjVu | .ps.gz ]

02/22: applications of gradient-based supervised learning

Subjects treated: Face detection, and other applications

Slides: face detection [DjVu].

02/24: a primer on probability theory

Subjects treated: distributions, marginalization, joint, conditional, exponential family, Gaussians.

Slides: [DjVu | PDF | PS]

03/01: intro to unsupervised learning

Subjects treated: density estimation. Maximum likelihood estimation, Gaussian estimation, Parzen Windows.

Slides: [DjVu | PDF | PS]

03/03: Dimensionality Reduction

Subjects treated: principal component analysis

Slides: [DjVu | PDF | PS]

03/08: Clustering, Data Compression

Subjects treated: K-Means clustering, vector quantization, image compression.

Slides: [DjVu | PDF | PS]

03/10: Expectation-Maximization

Subjects treated: latent variables, Expectation-Maximization algorithm (EM), mixtures of Gaussians.

Slides: [DjVu | PDF | PS]



03/22: Ensemble Methods, Boosting

Subjects treated: Ensemble methods, Jackknife, Bagging, Boosting.

Slides: [DjVu | PDF | PS]

HOMEWORK 09a: speech recognition with backpropagation and multilayer nets. Click here to get the text of the homework (caution: the file is 11MB).

The homework is due 03/29 before the class.

03/24: Bayesian Learning

Subjects treated: Bayesian Learning Methods.

Slides: [DjVu | PDF | PS]

03/29: Fast Optimization Methods

Subjects treated: Optimization, Hessian, Convergence of gradient descent, Newton's algorithm, Levenberg-Marquardt, Conjugate Gradient.

Required reading: "Efficient Backprop" [DjVu] [PS.GZ] [PDF]

03/31: Distributions on Strings

Subjects treated: Distributions on strings, weighted finite-state machines, finite-state transducers.

Slides: Tutorial by Mehryar Mohri and Michael Riley: [DjVu | PDF | PS]

HOMEWORK 10b: Principal Component Analysis Click here to get the text of the homework

The homework is due 04/12 before the class.

04/05: Introduction to Graphical Models

The lecture will be given by Marc'Aurelio Ranzato and Chris Poultney. Subjects treated: Graphical models, factorized probability distributions, belief propagation and the sum-product algorithm.

HOMEWORK 11a: K-Means Clustering Click here to get the text of the homework

The homework is due 04/14 before the class.

04/07: Lab: PCA, K-Means

Instructor: Raia Hadsell.

Subjects treated: implementing PCA and K-Means with applications to image compression.

04/12: Learning with Sequences

Subjects treated: Graphs Transformer Networks, sequence labeling, discriminative training.

Required Reading:

Gradient-based Learning Applied to Document Recognition by LeCun, Bottou, Bengio, and Haffner; pages 18 (sectionC) to 40. [DjVu | .ps.gz ]

HOMEWORK 12a: Mixture of Gaussians Click here to get the text of the homework

The homework is due 04/19 before the class.

04/14: Lab: gblearn2 library and convolutional nets

Instructor: Raia Hadsell.

Subjects treated: Training a handwritten digit recognizer.

FINAL PROJECT: most of the proposed topics of final projects will be related to this lab session. You can make a proposal for a different subject outside of the proposed list if you want. Proposed subjects will include various ways of training various architectures on the MNIST dataset of handwritten digits. Students will have 15 minutes to present the results of their project to the class in an oral exam session on Thursday, May 5 from 4:00 PM to 5:30 PM. You can do the project by yourself or in groups of two.