CSCI-UA.0202: Lab setup and tools

Overview

The main tools in this class are an x86 emulator, called QEMU, and a compiler toolchain (including assembler, linker, loader, C compiler, debugger), which will produce code (often a small-scale operating system) that will run on top of QEMU. These tools are included on the platform on which you will do your lab work. You have two choices for this platform:

  1. A virtual devbox that we provide. Instructions
  2. Machines provided by CIMS (The Courant Institute), on which you all have been given accounts. Instructions

The choice is up to you; there is no right answer. Each option has advantages and disadvantages. The advantage to the first option is that the lab will run completely on your own computer (and does not require Internet access, provided that you have the lab description locally). A disadvantage is that you have to do a bit more setup (which we describe and walk through below). A second disadvantage is that you will be interacting with two layers of virtualization (the host of the devbox, and then QEMU); this could be confusing. (This could also harm performance, but that will not be a concern for most of the labs.) The second option is the other way around: after your account is set up, you have to do very little setup, and there will be only one layer of virtualization. The disadvantage to this option is that you will be interacting with a computer in a lab somewhere, which requires either going to the lab, or working over ssh (which in turn requires Internet access).

There is a third option, but you're on your own if you choose it:

  1. Your own machine, without virtualization. We cannot support this option, and your code will be graded on the two platforms above. Instructions

Option 1: The virtual devbox

One way to ensure that we are all using a uniform development environment (short of us all using the same machine) is for us to use identically configured virtual machines. You can think of a virtual machine as a way to run a particular operating system (in our case, an Ubuntu desktop) on top of another operating system (the one that controls your laptop or desktop).

To get our virtual devbox running, you should do the following:

  1. Download and install on your computer (e.g., your desktop or laptop) a virtualization platform:
  2. Download to your computer the virtual devbox image that we have set up for you. Note: the file is 1.5 GB, so the initial download may take some time.
  3. Using the virtualization platform (for example, VirtualBox), import the image that you saved. You can leave all of the settings to their default values. This, too, may take some time.
  4. Using the virtualization platform, start up the virtual machine. You should see an Ubuntu desktop come online.
  5. Login to the machine. Username: cs202. Password: labs.
  6. At this point, you have a machine that will act like an Ubuntu desktop, and you'll do your development on this machine.
  7. If you are using VirtualBox, you will probably want to enlarge the default "monitor size" of the virtual machine. For this to work, the scaling feature must be enabled; this may require pressing a command key plus C, depending on your platform.

Notice the potential confusion from this arrangement. VirtualBox is creating a fake machine (with a fake x86 CPU). Ubuntu desktop is running on that fake machine (though neither Ubuntu desktop nor the applications (compiler, editor, browser) running on Ubuntu desktop are aware that they are running on a fake machine). The confusion enters because one of the applications that you are going to run on Ubuntu desktop is QEMU, and what does QEMU do? It creates another fake machine on which one can run an operating system. In our case, this fake machine will run the lab OSes (the mini operating systems and, perhaps, Linux). To recap: our lab OSes will run on the fake machine that QEMU provides, QEMU is running on Ubuntu desktop, Ubuntu desktop is the operating system for the fake machine that VirtualBox provides, and VirtualBox is running on your computer's operating system (known as the "host" operating system).

All of this is to say that there are actually two levels of virtualization in play. The two levels serve different purposes. The purpose of the lower is to create a uniform devbox. The purpose of the upper is to create the illusion of a physical machine so that you can develop, debug, and run the lab code. (This is a good example of how virtualization serves multiple purposes.)

A potentially interesting point is that although the two levels of virtualization serve different purposes, and although they are implemented with different software (VirtualBox and QEMU, respectively), they accomplish their function using similar technology. We discuss this technology in depth in advanced OS classes. In the meantime, welcome to The Matrix....

Option 2: CIMS machines

If you were registered for this class before Monday, January 26, then, thanks to the CIMS sysadmins, you have an account on machines administered by CIMS (short for Courant Institute of Mathematical Sciences, which is the home of the CS department). Thank you CIMS sysadmins!! If you added this class after January 26, and you have never had a CIMS account before (you can determine whether you had an account by going here), then please email our course's staff email alias (the address is on our home page). In this email, explain that you registered after January 26; in that case, we will close the loop between you and the CIMS sysadmins. If you were registered before January 26, and you cannot login, do not email us about that; we will probably drop such emails, since the instructions for this case are given below.

A few words about this account: it is independent of your NYU NetID/home account. This account will be valid during the semester, and will give you access to all of CIMS's computing resources.

If you have never had a CIMS account before (roughly half of you), then you have received an e-mail at your nyu.edu email address with setup instructions.

If you have had a CIMS account before (the other half of you), then you did not receive any kind of notification; you should simply login, following the instructions below. If you do not remember your password, or cannot login for other reasons, then go here.

There are two options for using CIMS computing resources. They are described below.

Computer lab

You can use the desktop machines in the Computer Labs in Rooms 229 and 230 Warren Weaver Hall. Here is more information about these Labs. You probably want the Linux machines. Type your CIMS username and password to login. After you login, you are ready to start the labs.

Remote Access

If you are not on campus, then you can access both the desktops and a collection of servers that CIMS maintains, using ssh. We describe the steps below. At a high level, you will be logging into a machine called access.cims.nyu.edu, and from there you will login to the CIMS machine of your choice. Please do not skip that second part, meaning run your lab on access.cims.nyu.edu; this will be viewed as antisocial.

Follow these steps:
  1. Gain access to the CIMS machines through ssh. The Secure Shell (SSH) establishes an authenticated, encrypted session with a designated machine. You probably want to become comfortable with ssh, if you are not already. A basic guide for interacting with ssh is available here. For more in-depth information, check here.

    For all users, if you have successfully logged in, you will see a banner like this:

    	############################################################################
    	#                                                                          #
    	#    CIMS Access Server                                                    #
    	#                                                                          #
    	#    ** Unauthorized access is prohibited by law! **                       #
    	#                                                                          #
    	#    Please do not run CPU-intensive jobs on this server.  For             #
    	#    information regarding appropriate systems on which to run such        #
    	#    processes, see:                                                       #
    	#                                                                          #
    	#           http://cims.nyu.edu/u/computeservers                           #
    	#                                                                          #
    	#    Users requiring a Solaris SPARC environment can login to              #
    	#    crunchy12.cims.nyu.edu                                                #
    	#                                                                          #
    	#    If you have any questions, please send mail to:                       #
    	#            helpdesk@cims.nyu.edu                                         #
    	#                                                                          #
    	############################################################################
    	
  2. Consult this list to choose a server. Then, from access.cims.nyu.edu, type the following command:
    $ ssh -X [username]@[server].cims.nyu.edu

    Be sure to use the -X flag again. Once you log on, you can test to make sure X forwarding is working with the following command:

    $ xclock &
    (A window should pop up with a simple clock.)

    You should also see your home directory on any of the machines.

  3. Optional but recommended: set up "ssh keys", which would remove some of the hassle of typing your password every time you ssh to a machine. For this, see the tutorials above, or consult one of the TAs.

You are now ready to begin the labs.

Option 3 (unsupported): Using your own machine, without virtualization

If you really want to, you can build and install the tools directly on your own machine. Howver, we cannot support this option, so you are on your own, for the most part. Furthermore, for the purposes of grading, your code must run on the two platforms above, so you should still test on one of them.

Below we give instructions for Unix and OS X computers. It should also be possible to get the development environment running under windows with the help of Cygwin. Install cygwin, and be sure to install the flex and bison packages (they are under the development header).

Step 1: Build and install the QEMU emulator

We recommend you use our (really, MIT's) patched version of QEMU instead of the stock version that may come with your distribution. To build your own patched version of QEMU:

  1. Clone the QEMU git repository from the MIT 6.828 people:
    git clone https://github.com/geofft/qemu.git -b 6.828-1.7.0
  2. You may need to install the SDL and pixman development libraries to get a graphical VGA window. On Debian/Ubuntu these are libsdl1.2-dev and libpixman-1-dev:
    sudo apt-get install libsdl1.2-dev libpixman-1-dev
  3. Configure the source code
    1. Linux: ./configure --disable-kvm [--prefix=PFX] [--target-list="i386-softmmu x86_64-softmmu"]
    2. OS X: ./configure --disable-kvm --disable-sdl [--prefix=PFX] [--target-list="i386-softmmu x86_64-softmmu"] The prefix argument specifies where to install QEMU; without it QEMU will install to /usr/local by default. The target-list argument simply slims down the architectures QEMU will build support for.
  4. Run make && make install This step may require privilege (sudo make install)

Any build problems are likely due to missing dependencies: be sure your system has the above libraries installed as well as normal development tools such as gcc, make, etc.

Step 2: Set up the compiler toolchain

Modern Linux and BSD UNIX distributions already provide a toolchain suitable for this course. To test your distribution, try the commands below. If you do not have the necessary tools present, you can either install the required packages from your distro's repository or build them from source.

Testing your existing compiler toolchain

To test your existing distribution, try the following commands:

$ objdump -i

The second line should say elf32-i386.

$ gcc -m32 -print-libgcc-file-name

The command should print something like /usr/lib/gcc/i486-linux-gnu/version/libgcc.a or /usr/lib/gcc/x86_64-linux-gnu/version/32/libgcc.a

If both these commands succeed, you're all set, and don't need to install your own toolchain.

Installing the compiler toolchain from packages

The commands below are to install the compiler toolchain from Ubuntu/Debian's package repository. (If you're using a non-Debian based distro, the package names should be similar, but the installation commands will be different. If you're using OS X, try using the homebrew package manager or build from source.)

$ sudo apt-get install build-essential gcc-multilib gdb

Building the compiler toolchain from source

You can use your own toolchain. Note that this may require editing the makefiles for the labs to include the tools that you are building. For example, in the WeensyOS labs, you would add the following line to the file conf/env.mk in the lab code:

GCCPREFIX=

We assume that you are installing the toolchain into /usr/local; below, we give instructions for overriding this. You will need a fair amount of disk space to compile the tools (around 1GB). If you don't have that much space, delete each directory after its make install step.

Download the following packages:

(NOTE: These instructions may be out of date, but they should give you a sense of how to build your toolchain from source.)

(You may also use newer versions of these packages.) Unpack and build the packages. This text shows you how to install into /usr/local, which is what we recommend. To install into a different directory, $PFX, click here (hide). If you have problems, see below.

export PATH=$PFX/bin:$PATH export LD_LIBRARY_PATH=$PFX/lib:$LD_LIBRARY_PATH # On OS X, use DYLD_LIBRARY_PATH
tar xjf gmp-5.0.2.tar.bz2 cd gmp-5.0.2 ./configure --prefix=/usr/local--prefix=$PFX make make install # This step may require privilege (sudo make install) cd .. tar xjf mpfr-3.0.1.tar.bz2 cd mpfr-3.0.1 ./configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX make make install # This step may require privilege (sudo make install) cd .. tar xzf mpc-0.9.tar.gz cd mpc-0.9 ./configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX make make install # This step may require privilege (sudo make install) cd .. tar xjf binutils-2.21.1.tar.bz2 cd binutils-2.21.1 ./configure --prefix=/usr/local--prefix=$PFX --target=i386-jos-elf --disable-werror make make install # This step may require privilege (sudo make install) cd .. i386-jos-elf-objdump -i # Should produce output like: # BFD header file version (GNU Binutils) 2.21.1 # elf32-i386 # (header little endian, data little endian) # i386... tar xjf gcc-core-4.6.1.tar.bz2 cd gcc-4.6.1 mkdir build # GCC will not compile correctly unless you build in a separate directory cd build ../configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX --with-mpc=$PFX \ --target=i386-jos-elf --disable-werror \ --disable-libssp --disable-libmudflap --with-newlib \ --without-headers --enable-languages=c
MAC OS X 10.7 "LION" NOTE: The default clang compiler on OS X 10.7 cannot build a working version of GCC. Use the following configure line to work around the problem:
../configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX --with-mpc=$PFX \ --target=i386-jos-elf --disable-werror \ --disable-libssp --disable-libmudflap --with-newlib \ --without-headers --enable-languages=c \ CC=/usr/bin/gcc-4.2 CPP=/usr/bin/cpp-4.2 \ make all-gcc make install-gcc # This step may require privilege (sudo make install-gcc) make all-target-libgcc make install-target-libgcc # This step may require privilege (sudo make install-target-libgcc) cd ../.. i386-jos-elf-gcc -v # Should produce output like: # Using built-in specs. # COLLECT_GCC=i386-jos-elf-gcc # COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/i386-jos-elf/4.6.1/lto-wrapper # Target: i386-jos-elf tar xjf gdb-7.3.1.tar.bz2 cd gdb-7.3.1 ./configure --prefix=/usr/local--prefix=$PFX --target=i386-jos-elf --program-prefix=i386-jos-elf- \ --disable-werror make all make install # This step may require privilege (sudo make install) cd ..

Troubleshooting option 3

Q. I can't run make install because I don't have root permission on this machine.
A. Our instructions assume you are installing into the /usr/local directory. However, this may not be allowed in your environment. If you can only install code into your home directory, that's OK. In the instructions above, replace --prefix=/usr/local with --prefix=$HOME (and click here (hide) to update the instructions further). You will also need to change your PATH and LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (Mac OS X) environment variables, to inform your shell where to find the tools. For example:
export PATH=$HOME/bin:$PATH
export LD_LIBRARY_PATH=$HOME/lib:$LD_LIBRARY_PATH
Enter these lines in your ~/.bashrc file so you don't need to type them every time you log in.
Q. My build fails with an inscrutable message about "library not found".
A. You need to set your LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (OS X) environment variable to include the libraries you built; see above. The environment variable must include the PREFIX/lib directory (for instance, /usr/local/lib).
Q. When building the labs, linking fails with error messages like "__udivdi3 not found" and "__muldi3 not found".
A. To build 32-bit binaries on 64-bit machines, gcc-multilib is required. On Ubuntu Linux, try this to fix the problem: sudo apt-get install gcc-multilib

(Adapted from MIT's 6.828 Tools page.)


Last updated: Sun Nov 15 21:08:41 -0500 2015 [validate xhtml]