This page contains the schedule, slide from the lectures, lecture notes, reading lists,
assigments, and web links.
I urge you to download the DjVu viewer
and view the DjVu version of the documents below. They display faster,
are higher quality, and have generally smaller file sizes than the PS and PDF.
Fulltext search is provided for the entire
collection of slides and papers. Click here to search
You can have a look at the schedule
and class material for the graduate version of this course taught during the
Fall 2004 semester, but be warned that the undergraduate edition is
significantly different.
01/18: Introduction and basic concepts 
Subjects treated: Intro, types of learning, nearest neighbor, how biology does it,
linear classifier, perceptron learning procedure, linear regression,
Slides: [DjVu  PDF  PS]
01/20: Introduction to Lush, Computer Lab 
IMPORTANT
NOTE: This session will start in the usual classroom WWH102,
but the second part will be in room WWH512
(Sun computer lab on the 5th floor).
Subjects treated: An introduction to the
Lush programming language.
Implementing the perceptron learning algorithm.
The instructor for this lab seesion will be FuJie Huang (Jie).
You should all have a login on the Sun machines in the lab.
If you have laptop, you are welcome to install Lush on it
and to use it for the computer lab sessions instead of the
labs's Sun workstations. Lush runs on Linux, Mac OSX,
Solaris, and on Windows under Cygwin.
Click here to get the files for the lab
and the homework.
HOMEWORK 01b: The homework is due 02/08 before the class (NOTE: new
due date).
Slides: [DjVu  PDF  PS]
Lush Tutorial: read the short tutorial on Lush at
the Lush documentation page.
01/25: Perceptron, Linear Regression 
Subjects treated: Intro, perceptron convergence theorem,
multivariate calculus refresher, loss functions, linear regression, LMS/Adaline.
Slides: [DjVu  PDF  PS]
02/27: EnergyBased Models, Loss Functions, Linear Regression 
Subjects treated: EnergyBased Models, Inference,
Loss Functions, Logistic Regression.
Slides: [DjVu  PDF  PS]
02/01: Lab: Linear Classifiers 
IMPORTANT NOTE:
We will meet in the usual classroom WWH102,
and promptly go to the Sun computer lab in room WWH512.
.
Subjects treated: LAB: perceptron, LMS, Logistic Regression.
HOMEWORK 02a: Click here to get the files
for the lab and the homework.
To uncompress the homework file on Unix/Linux do
"tar xvfz lab02a.tgz". This will create a directory
named "lab02a". On Windows, use Winzip.
The homework is due 02/15 before the class.
02/03: Generalization, Regularization 
Subjects treated:
Learning and Generalization, Regularization.
Slides: [DjVu  PDF  PS]
02/08: MultiModule Systems, Backpropagation 
Subjects treated: Multilayer and MultiModule Systems,
Gradient BackPropagation.
Slides: [DjVu  PDF  PS]
02/10: Modules and Architectures 
Subjects treated: Modules and Architectures.
Slides: [DjVu  PDF  PS]
02/15: Special Architectures 
Subjects treated: Special Architectures for
Time Series, Images, Audio, Video.
Slides: [DjVu  PDF  PS]
02/17: Audio and Image Classification 
Subjects treated: Convolutional Networks for
speech and image classification.
Slides: object recognition [DjVu].
HOMEWORK 05b: computing Jacobians. Click one these links
to get the text of the homework: [DjVu  PDF  PS]
The homework is due 03/01 before the class.
Required Reading:
Gradientbased Learning Applied to Document Recognition by LeCun,
Bottou, Bengio, and Haffner; pages 1 to the first column of page 18:
[DjVu  .ps.gz ]
02/22: applications of gradientbased supervised learning 
Subjects treated: Face detection, and other applications
Slides: face detection [DjVu].
02/24: a primer on probability theory 
Subjects treated: distributions, marginalization, joint,
conditional, exponential family, Gaussians.
Slides: [DjVu  PDF  PS]
03/01: intro to unsupervised learning 
Subjects treated: density estimation. Maximum likelihood
estimation, Gaussian estimation, Parzen Windows.
Slides: [DjVu  PDF  PS]
03/03: Dimensionality Reduction 
Subjects treated: principal component analysis
Slides: [DjVu  PDF  PS]
03/08: Clustering, Data Compression 
Subjects treated: KMeans clustering, vector quantization,
image compression.
Slides: [DjVu  PDF  PS]
03/10: ExpectationMaximization 
Subjects treated: latent variables,
ExpectationMaximization algorithm (EM), mixtures of Gaussians.
Slides: [DjVu  PDF  PS]
03/22: Ensemble Methods, Boosting 
Subjects treated: Ensemble methods,
Jackknife, Bagging, Boosting.
Slides: [DjVu  PDF  PS]
HOMEWORK 09a: speech recognition with
backpropagation and multilayer nets.
Click here to get the text
of the homework (caution: the file is 11MB).
The homework is due 03/29 before the class.
Subjects treated: Bayesian Learning Methods.
Slides: [DjVu  PDF  PS]
03/29: Fast Optimization Methods 
Subjects treated: Optimization, Hessian,
Convergence of gradient descent, Newton's algorithm,
LevenbergMarquardt, Conjugate Gradient.
Required reading: "Efficient Backprop"
[DjVu]
[PS.GZ]
[PDF]
03/31: Distributions on Strings 
Subjects treated: Distributions on strings,
weighted finitestate machines, finitestate transducers.
Slides: Tutorial by Mehryar Mohri and Michael Riley: [DjVu  PDF  PS]
HOMEWORK 10b: Principal Component Analysis
Click here to get the text
of the homework
The homework is due 04/12 before the class.
04/05: Introduction to Graphical Models 
The lecture will be given by Marc'Aurelio Ranzato and Chris Poultney.
Subjects treated: Graphical models, factorized probability
distributions, belief propagation and the sumproduct algorithm.
HOMEWORK 11a: KMeans Clustering
Click here to get the text
of the homework
The homework is due 04/14 before the class.
Instructor: Raia Hadsell.
Subjects treated: implementing PCA and KMeans with
applications to image compression.
04/12: Learning with Sequences 
Subjects treated: Graphs Transformer Networks, sequence
labeling, discriminative training.
Required Reading:
Gradientbased Learning Applied to Document Recognition by LeCun,
Bottou, Bengio, and Haffner; pages 18 (sectionC) to 40.
[DjVu  .ps.gz ]
HOMEWORK 12a: Mixture of Gaussians
Click here to get the text of the homework
The homework is due 04/19 before the class.
04/14: Lab: gblearn2 library and convolutional nets 
Instructor: Raia Hadsell.
Subjects treated: Training a handwritten digit recognizer.
FINAL PROJECT: most of the proposed topics of final projects
will be related to this lab session. You can make a proposal for a
different subject outside of the proposed list if you want. Proposed
subjects will include various ways of training various architectures
on the MNIST dataset of handwritten digits. Students will have 15
minutes to present the results of their project to the class in an
oral exam session on Thursday, May 5 from 4:00 PM to 5:30 PM. You can
do the project by yourself or in groups of two.
