Physical Media Projects

The primary goal of our project is to combine music and video capturing techniques. We want to propose a toy for kids, especially for kids are curious about everything, and it¡¦s easy and cheap to get entertained from it. We hope we can stimulate their music talent.

This program gets the input from the video camera. The player doesn¡¦t use any traditional keyboard or mouse as an input device. He/she just uses anything in hand to setting as an input, then the program will get the signal from it. This project provides three instruments, piano, conga drum, and salsa maracas. We can select one of these instruments you want to accompany with the main theme. While the main theme plays, the player can use any pre-setting device as an input to interact with our program. For instance, if the player selects conga drum icon, then there will have two drum icons on the screen. As the input device waves through one of these two icons, it will make the drum sound. So the player can play the main theme first, and then make the sound of drum as he wants to create a new rhythm. We also provide the recording feature to record the sounds and the main theme, and the player can playback to check if you make a masterpiece!!

In this project, there are two major technical knowledge, video process and music mixer. About video capture, we need to process the images that we get from video camera. As we get the image, we transfer it to frame buffer, and analyze every pixel in this image. Then we identify the input device and its location in the image to determine the response. About music mixer, we saved each sound as a wav file. If some input get in, then we play the file to make sound. Moreover, we implement an recorder program to record the sound that mix the main theme and drum or salsa maracas sound, and a player to enable the player to listen the new rhythm that just created.

We select Java as our programming language. Because Java provides a framework called JMF that includes bunch of video processing library for us, it reduces the complexity of video processing and saves plenty of work for us.

We implement an embeded system with LEDs and motors by Arduino. By programming the "Processing", we manipulate signal sent the components, like make the LEDs shimmer. There are four motors used to trigger the icons up and down. We made up the device with eight gears can spin to trigger the action.