
This page contains the schedule, slide from the lectures, lecture notes, reading lists,
assigments, and web links.
I urge you to download the DjVu viewer
and view the DjVu version of the documents below. They display faster,
are higher quality, and have generally smaller file sizes than the PS and PDF.
Fulltext search is provided for the entire
collection of slides and papers. Click here to search
You can have a look at the schedule
and class material for the version of this course taught during the
Spring 2004 semester, but be warned that the new edition is
significantly different.
09/07: Introduction and basic concepts 
Subjects treated: Intro, types of learning, nearest neighbor, how biology does it,
linear classifier, perceptron learning procedure, linear regression,
Slides: [DjVu  PDF  PS]
Recommended Reading:
 Hastie/Tibshirani/Friedman: Chapter 2
 Refresher on random variables and probabilites by
Andrew Moore: (slides 127) [DjVu  PDF]
 Refresher on joint probabilities, Bayes theorem by
Chris Willams: [DjVu  PDF]
 Refresher on statistics and probabilities by
Sam Roweis: [DjVu  PS]
 If you are interested in the early history of selforganizing
systems and cybernetics, have a look at this book available from the
Internet Archive's Million Book Project: SelfOrganizing
Systems, proceedings of a 1959 conference edited by Yovits and
Cameron (DjVu viewer required for full text).
09/14: EnergyBased Models, Loss Functions, Linear Machines 
Subjects treated: Energybased models, minimumenergy
machines, loss functions. Linear machines: perceptron, logistic
regression. Linearly parameterized classifiers: Polynomial
classifiers, basis function expansion, RBFs, Kernelbased expansion.
Slides: [DjVu  PDF  PS]
09/21: GradientBased Learning I, MultiModule Architectures and BackPropagation 
Subjects treated: MultiModule learning machines. Vector
modules and switches. Multilayer neural nets. Backpropagation
Learning. Intro to Model Selection, structural risk minimization, regularization.
Slides on Regularization: [DjVu  PDF  PS]
Slides on MultiModule BackPropagation: [DjVu  PDF  PS]
Required Reading:
Gradientbased Learning Applied to Document Recognition by LeCun,
Bottou, Bengio, and Haffner; pages 1 to the first column of page 18:
[DjVu  .ps.gz ]
09/28: GradientBased Learning II: Special Modules and Architectures 
Subjects treated: Trainers; complex topologies; special
modules; Crossentropy and KLdivergence; RBFnets, Mixtures of
Experts; Parameter space transforms; weight sharing; convolution
module; TDNN; Recurrent nets.
Slides: [DjVu  PDF  PS]
Homework Assignements 01: implementing the Perceptron
Algorithm, MSE Classifier (linear regression), Logistic Regression.
Details and datasets below:
 Download this tar.gz archive. It
contains the datasets and the homework description.
 Decompress it with "tar xvfz homework01.tgz" on Unix/Linux or
with Winzip in Windows.
 The file homework01.txt contains the questions and instructions.
 Most the of the necessary Lush code is provided.
 Due Date is Tuesday October 19th, before the lecture.
10/05: Convolutional Nets, Image Recognition, Convergence and Optimization 
Subjects treated: Convolutional Networks; Image recognition,
object detection, and other applications; Convergence of
gradientbased optimization and acceleration techniques.
Slides: talk on object recognition with convolutional nets: DjVu
Slides on optimization: [DjVu  PDF  PS]
Required Reading:
If you haven't read it already: Gradientbased Learning Applied to
Document Recognition by LeCun, Bottou, Bengio, and Haffner; pages 1 to
the first column of page 18:
[ DjVu  .ps.gz ]
Optional Reading: FuJie Huang, Yann LeCun, Leon Bottou: "Learning Methods for Generic Object
Recognition with Invariance to Pose and Lighting.", Proc. CVPR 2004.
.ps.gz
NO LECTURE
Required Reading:
10/19: Bayesian Learning, MLE, MAP 
Subjects treated: Refresher probability theory;
Bayesian Estimation, Maximum Likelihood Estimation, Maximum A
Posteriori Estimation, Negative LogLikelihood Loss Functions.
Slides: Refresher on Probability Theory: [DjVu  PDF  PS]
Slides: Bayesian Learning: [DjVu  PDF  PS]
Required Reading:
Homework 01 due TODAY!
10/26: Unsupervised Learning 
Subjects treated: Unsupervised Learning: Principal Component
Analysis. Density Estimation: Parzen Windows, Mixtures of Gaussians,
AutoEncoders. Latent variables. Intro to the EstimationMaximization algorithm.
Slides:
11/02: Efficient Optimization, Latent Variables, Graph Transformer Networks 
Subjects treated:
Modeling distributions over sequences. Learning machines that
manipulate graphs. Finitestate transducers. Graph Transformer
Networks.
Efficient learning: Newton's algorithm, LevenbergMarquardt.
Required Reading:
Note: the slides used in class are not provided because the two
following papers cover the material.
Homework Assignements: implementing GradientBased Learning
and backpropagation. You must implement gradientbased learning using
the objectoriented, modulebased approach as described in class.
Various architectures, including a multilayer neural net, must be
implemented and tested on two datasets.
 Download this tar.gz archive. It
contains the datasets and the homework description.
 Decompress it with "tar xvfz homework02.tgz" on Unix/Linux or
with Winzip in Windows.
 The file homework02.txt contains
the questions and instructions.
 Most of the necessary Lush code is provided.
 Due Date is Friday Nov 19.
11/09: ExpectationMaximization, Hidden Markov Models I 
Subjects treated:
More on optimization methods for lerning: GaussMewton,
LevenbergMarquardt, BFGS, Conjugate Gradient;
ExpectationMaximization Algorithm (EM).
Introduction to Hidden Markov Models (HMM).
Required Reading:
11/16: HMM, Learning Theory, Bagging Boosting, VCDim 
Subjects treated: HMM learning.
Ensemble methods, Full Bayesian Learning, Bagging, Boosting.
Learning Theory, Bounds, VCDimension.
Slides:
Homework Assignements: Homework 03: KMeans and Mixture of Gaussians estimation with EM.
 The subject of this homework is to implement the Kmeans algorithm
and the ExpectationMaximization algorithm for a Mixture of Gaussians model.
The algorithms must be tested on image data for simulated image
compression taks.
 Download this tar.gz archive. It
contains the datasets and the homework description.
 Decompress it with "tar xvfz homework03.tgz" on Unix/Linux or
with Winzip in Windows.
 The file homework03.txt contains
the questions and instructions.
 DUE DATE: Friday Dec 3
11/23: Intro to Graphical Models 
Subjects treated: Intro to graphical models,
Belief Networks and Factor Graphs, Inference, Belief Propagation, Boltzmann Machines.
Homework Assignements: Final Project
 A list of possible project topics is
available here.
You are welcome to pick from this list of to propose a
project of your own (possibly in line with your main research interests).
To make a final project proposal, send a short description be
email to YLC and to the TA.
 This project will count for 40% of the final grade.
 Collaboration: you can do your final project in groups of two students.
 Due Date: If you need a grade right away (e.g. if you are
graduating this semester), you must turn in your final project by
December 17th.
 Extra Time: Extensions can be granted for ambitious projects by
students who are not graduating this semester. Send requests
for extensions to YLC.

