Graham Taylor

Modeling Human Motion Using Binary Latent Variables

Sample code provided by Graham Taylor, Geoff Hinton, and Sam Roweis

Permission is granted for anyone to copy, use, modify, or distribute this program and accompanying programs and documents for any purpose, provided this copyright notice is retained and prominently displayed, along with a note saying that the original programs are available from our web page.

The programs and documents are distributed without any warranty, expressed or implied. As the programs were written for research purposes only, they have not been tested to the degree that would be advisable in any important application. All use of these programs is entirely at the user's own risk.

Change Log

Instructions

  1. Create a separate directory and download mhmublv_code.zip into that directory
  2. Unzip the above file, preserving directory structure and filenames
  3. Download the sample data, data.mat and save it in the "mhmublv/Data" subdirectory that was created during the previous step
  4. To perform learning and generation, run "motiondemo.m" in Matlab
  5. The file, README.txt provides more details

Acknowledgments

The sample data we have included has been provided by Eugene Hsu:
http://people.csail.mit.edu/ehsu/work/sig05stf/

Several subroutines related to motion playback are adapted from Neil Lawrence's Motion Capture Toolbox:
http://www.cs.man.ac.uk/~neill/mocap/

Several subroutines related to conversion to/from exponential map representation are provided by Hao Zhang:
http://www.cs.berkeley.edu/~nhz/software/rotations
(Note: as of August 2008, it appears that Hao has moved and this link is no longer working)

Notes/FAQ

How did you go from Eugene Hsu's data.zip to the data structures skel,Motion in data.mat?

The MIT data has been obtained from Eugene Hsu (see link above). There is an archive here, data.zip including several motions in txt format.

The data.mat file I provide is only a portion of the sequence Normal1_M.txt from this dataset.

Here's how I have generated the {skel, Motion} data structures in data.mat. In Matlab:

M = dlmread('insert_your_path_here/Normal1_M.txt','\t');

This reads the 108 columns in this tab-delimited text file into the Matrix, M. Each row represents a frame.
It also reads in a blank column, 109. So I discard it.

M(:,109)=[];

The meaning of the columns are described in the readme.txt in data.zip. You could replace Normal1_M.txt with any of the other text files.

To play this motion, with my player, you will need a skeleton. You can use the "skel" structure in my data.mat file. There is also a script in the Motion/ subdirectory called buildskel_mit.m which demonstrates how "skel" is built.

The skeleton is not part of the data on which we train the model, but we need it to play back the motion.

It describes the hierarchy of segments, and which segments correspond to which columns of the data matrix.

Note that you can play the motion with expPlayData.m:

figure(2); expPlayData(skel,M,1/120);

Note that the data is at 120fps and it is very big so this will take a long time.
This data is extremely noisy, so I have isolated some "clean" segments.

Motion{1}=M(551:2300,:);
Motion{2}=M(2621:3660,:);
Motion{3}=M(3791:163000,:);

I have not touched the data in any other way.

How can I use data from the CMU motion capture database with your code?

My scripts are compatible with acclaim format (I have used CMU data which is in amc/asf). I use Neil Lawrence's toolbox (see link above) to load an acclaim asf and amc file.

Just a note: I noticed that recent versions of his toolbox were giving me strange results on older versions of Matlab. It works fine for me on Matlab 7.5 or 7.6, so just a warning if the routines are giving you weird errors.

The relevant scripts are acclaimReadSkel.m and acclaimLoadChannels.m (you need to call acclaimReadSkel.m first as the skel structure needs to be passed to acclaimLoadChannels).

So for each sequence, this would give you skel, channels. The "skel" should be the same for multiple sequences of the same subject. Then I stick all of the channels (of varying length) into the cell array called Motion (i.e. for all ii, call acclaimLoadChannels and then set Motion{ii}=channels)

Then to use my code, Motion will need to be converted into exponential map form. The relevant scripts are in the Motion/ subdirectory. Given that skel, Motion are now defined, batchConvertCMU.m will take each sequence in Motion and convert it to exponential maps via the function euler2expmap.m.

This will actually return skel1,Motion1 and so you should just set skel=skel1; Motion=Motion1; (as my code expects these variable names).

Now your data should be in the right format to work with my preprocessing scripts.