Skip to topic | Skip to bottom
Home
DRAGON
National
Science
Foundation
Register or Logon
Private Intranet

Start of topic | Skip to actions

Running DRAGON Software under User-Mode Linux (UML)

Contents:

Overview

DRAGON's GMPLS Control Plane software is intended to be run on multiple Unix PCs across a network. The software uses raw IPv4 sockets to communicate to neighboring network elements via GRE/IPIP/IPsec tunnels (for OSPF adjacencies and RSVP signaling), so multiple instances cannot be run on a single operating system. Not much can be done with a single instance of the DRAGON code running on a single machine, so using user-mode linux was investigated as a possible solution to this problem.

It is now possible to run DRAGON's GMPLS Control Plane software in a user-mode linux environment, allowing an entire GMPLS network to be simulated on a single Linux PC. This page is intended to contain all of the information a user would need to setup such an environment. Please note that only the GMPLS control plane is currently emulated, we do not yet simulate a data plane. More information about this can be found in the section on future work.

For background information on User-Mode Linux, we recommend visiting the User-mode Linux Kernel Home Page or by reading Jeff Dike's book on User-Mode Linux. There is also a whitepaper here on Advanced Network Simulation under User-Mode Linux which is exploring many similar concepts.

Virtual Network Experiments (VNE)

An existing open-source package called Virtual Network Experiments (VNE) has been slightly modified to be able to run the DRAGON code in the UML environment, in such a way that an XML file can be used as to express the desired topology of the simulated GMPLS network. A paper describing VNE can be found at http://www.os3.nl/~gjv/uml/anp-jvdh-gjv.pdf or refer to the mirrored copy. Example XML files are included below to allow the user to quickly bring up simulated GMPLS networks of varying levels of complexity. The VNE software was originally developed by the Universiteit of Amsterdam's Systems and Networking Engineering research group. A webpage for VNE has been setup at http://ndl.uva.netherlight.nl/trac/vne/

VNE overview graphic

Requirements

  • any x86-based PC that is capable of running Linux
    • a minimum of 512MB of RAM in the UML host machine will be required to run the basic examples
    • 1GB of RAM (or more) is needed for larger, more complex simulations
    • each UML instance requires about 32MB of physical memory
  • any Linux distribution
    • our testing was done with the Debian 3.1 ("sarge"), Ubuntu 6.10 ("edgy eft") and Redhat Fedora Core 3 distributions
    • the latest debian ("unstable") should also be fine
    • any system running a Linux 2.4 or 2.6 kernel should work

The remaining prerequisites are available for download from this page:

  • Virtual Network Experiments (VNE) software package
  • UML kernel binary
  • custom UML root filesystem
    • includes DRAGON's GMPLS Control Plane Software and supporting utilities

Step-by-Step Installation

Configuration

  • This example assume the following structure:
    • location of root filesystem: /a/uml/debian31/Debian-3.1-x86-root_fs
    • location of UML kernel binary: /a/uml/debian31/linux

  • Create the file ~/.vnerc (in your home directory) that contains the following, changing any paths as needed for your system. The defaults are contained in a text file called VNEDefaults.cfg within the VNE distribution. 
[DEFAULT]
kernel     = /a/uml/debian31/linux
filesystem = /a/uml/debian31/Debian-3.1-x86-root_fs
fstype     = cow
xmllint    = /usr/bin/xmllint
screen     = /usr/bin/screen
xterm      = /usr/X11R6/bin/xterm
uml_switch = /usr/bin/uml_switch
memorysize = 32

  • Use which to find the location of the binaries on your system, e.g. which screen
    • default location for screen binary is /usr/bin on Debian/Ubuntu
      • Fedora/Redhat systems seem to use /usr/local/sbin/screen
    • default location for xterm binary is /usr/X11R6/bin on Debian/Ubuntu
      • Fedora/Redhat systems install a symlink from /usr/X11R6/bin/xterm to /usr/bin/xterm

  • If your system has a lot of memory, you can change memorysize to a higher number. This will allow you to perform development work, e.g. re-compiling the DRAGON software

  • Setup a symbolic link to UML kernel binary:
    • run a command such as ln -s /a/uml/debian31/uml_linux-2.6.20 /a/uml/debian31/linux
    • this makes experimenting with different UML kernels easy
    • optionally you could point .vnerc directly to the kernel

  • mkdir ~/uml
    • This is where the copy-on-write (COW) files will be placed, one for each UML instance
    • COW files contain changes/diffs to the read-only root filesystem (so they allow for a full read-write root partition in UML)
    • NOTE: This directory must exist in advance before running VNE! (TODO: fix this at some point)

Usage

  • ssh to UML host with X11 forwarding turned on
    • we highly recommend using compression, as it greatly reduces the bandwidth usage for the X11 forwarding
    • ssh -C -X user@hostname
    • make sure that you have X11Forwarding yes in the server's sshd_config file
  • Run VNE with some XML topology file as input
    • There are explanations and examples of how to provision an LSP inside the comments of each XML file: (located inside the VNE-dragon-snapshot distribution)
      • bridged_triangle.xml —
      • config-4domains.xml —
      • config-narb-inter-uni-csa.xml —
      • config-narb-intra-p2p-csa.xml —
      • config-narb-intra-uni-csa.xml —
      • config-stress.xml —
      • config.xml —
      • example.xml —
      • test_config.xml —
      • testconfig.xml —
    • Assuming you've used the suggested paths, it should be possible to do something like this: 
root@hydra:~# cd VNE-dragon
root@hydra:~/VNE-dragon# ./VNE config.xml
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Welcome to the text interface of Virtual Network Experiments!
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Important commands: help (or ?), shell (or !) and quit
Type 'help' or press <Tab> twice for a complete list of commands.

>

  • Try running the config command to dump the configuration loaded from the XML file: 
> config
------------------------------------------------------------------------------
                Configuration for es1 (Host)
------------------------------------------------------------------------------
es1: /a/uml/debian31/linux mem=128m con1=pts umid=es1 ubda=/home/dragon/uml/es1,/a/uml/debian31/Debian-3.1-x86-root_fs eth0=daemon,,unix,/tmp/tmpmA16Q4/uml_switch_socket
kernel: /a/uml/debian31/linux
filesystem: /a/uml/debian31/Debian-3.1-x86-root_fs (cow)
[...]

  • Or maybe list ints to list the network interfaces: 
> list ints
Interface eth0 on host es1: eth0 (ipv4Address=10.0.0.2, ipv4Netmask=255.0.0.0)
Interface eth0 on host es2: eth0 (ipv4Address=10.0.0.5, ipv4Netmask=255.0.0.0)
Interface eth0 on host switch1: eth0 (macAddress=00:00:AA:BB:00:01, part of bridge br0)
Interface eth1 on host switch1: eth1 (macAddress=00:00:AA:BB:00:02, part of bridge br0)
Interface eth2 on host switch1: eth2 (macAddress=00:00:AA:BB:00:03, part of bridge br0)
Interface eth3 on host switch1: eth3 (macAddress=00:00:AA:BB:00:04, part of bridge br0)
Interface eth0 on host vlsr1: eth0 (ipv4Address=10.0.0.3, ipv4Netmask=255.0.0.0)
Interface eth0 on host vlsr2: eth0 (ipv4Address=10.0.0.4, ipv4Netmask=255.0.0.0)

  • To really get things going, run the following commands and you should get control of the UML instances: 
> start all
> xterm all

  • Once the xterms are loaded, click on one, press enter and login as root.
    • By default, there is no password set.

  • In VNE's interactive CLI:
    • config — prints the configuration details to the console
    • start all or start host — start all (or one) of the UML instances
    • xterm all or xterm host — launch xterms for user interaction with all (or one) of the UML instances
    • stop all or stop host — stop all (or one) of the UML instances
    • quit — exit VNE

  • You don't have to use xterms to interact with the instances — you can also start another SSH session to the UML host and run screen -r vlsr1 to connect to the instance named vlsr1.

Screenshots

  • UML screenshot (using the config.xml example from the VNE package):
    UML screenshot (using the config.xml example from the VNE package)

Movies

  • Demo movie showing user running VNE and "pinging" over circuits using a simulated data plane:
    VNE Circuit Demo Movie w/ simulated data plane
    • Data plane is simulated using vconfig/brctl on Linux switching nodes to simulate hardware Layer 2 switches

Presentations

Advanced Topics

hostfs

hostfs is an alternative to using COW sparse files. Instead of using an image file for the root filesystem and COW sparse files for the diffs to that read-only filesystem, hostfs allows you to simply mount a directory from the UML host machine. This provides a convenient mechanism to get files in and out of UML instances, share data between instances, etc. It is even possible to make the root partition use hostfs, effectively using a directory on the UML host machine as the root filesystem for the UML instance. When using hostfs with the root partition, root has to be mounted read-only.

If you create a directory on the UML host such as /a/uml/hostfs, you can now access this directory from within the UML instance by running this command inside the instance: mount none /mnt -t hostfs -o /a/uml/hostfs

dumping GMPLS-RSVP-TE/GMPLS-OSPF-TE packets with tshark

tshark is installed in the root filesystem, the CLI version of Ethereal/Wireshark. If you are interested in dumping the RSVP and OSPF packets, run commands such as these on different GRE interfaces to see what is happening with these protocols: 

vlsr1:~# tshark -nV -i gre1 ip proto 46
vlsr1:~# tshark -nV -i gre1 ip proto 89

Notes on compiling your own UML kernel

We were not able to successfully compile the UML kernel under Ubuntu 6.10 (Edgy Eft), but we had no problems compiling a custom UML kernel on a Debian 3.1 machine. The custom kernel was then copied to the Ubuntu 6.10 machine, which was hosting the UML instances. The details about how to compile a custom UML kernel can be found on the User-Mode Linux Home Page: Compiling the kernel and modules page.

Below is a copy of the .config file used to build our custom UML kernel:

One potential issue with using the pre-compiled UML kernel supplied on this page is that it is dynamically linked, as opposed to statically linked: 

# file linux-2.6.18.3
linux-2.6.18.3: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs), for GNU/Linux 2.2.0, stripped

Since this UML kernel is dynamically linked, it relies on several other libraries being present: 

# ldd linux-2.6.18.3
        linux-gate.so.1 =>  (0xffffe000)
        libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7fcc000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e98000)
        /lib/ld-linux.so.2 (0xb7fd2000)

If you are using Debian 3.1 or Ubuntu 6.10, the pre-compiled UML kernel should work fine in your environment. We have not investigated how to compile a statically linked UML kernel, so please let us know if you know of a way to do that. A statically linked UML kernel should be more likely to work on other Linux distributions.

UML Kernel Debugging

kernel with debugging option turned on can be debugged in gdb, provide example of how to attach gdb to a running UML instance.

Modifying the root filesystem

We use the following script to make changes to the root filesystem. It is important to be sure that no UML instances are running while making modifications, and the copy-on-write (COW) files should also be deleted when making changes to the base filesystem. They will be recreated automatically the next time that you run VNE. 

% cat mount_fs_local.sh 
#!/bin/sh
echo make sure there are no linux processes below using the filesystem:
ps auxwww|grep linux
echo removing any existing copy-on-write COW files from ~/uml..
rm -vf ~/uml/*
mount -v /a/uml/debian31/Debian-3.1-x86-root_fs /mnt -o loop

Known Bugs

  • There were problems with hardcoded paths in to certain executables in older versions of VNE, but this has been taken care of with the ~/.vnerc file

  • The quit command in the VNE text interface may not properly halt the UML instances
    • reason: VNEDevice.py's deactivate() function is never called...?
    • run stop all to correctly halt all UML instances before running quit

  • Sometimes UML instances can freeze or become unresponsive or not exit cleanly...or VNE may crash for some reason, leaving stale instances around.
    • This is usually sufficient to clean things up:
      • killall linux; sleep 1; killall -9 linux and rm ~/uml/*, to remove all of the COW files
    • Instances may "hang" in tcsetattr():

  • fsck wants to run after a certain number of bootups, causing problems/delays..
    • You may need to delete all of the COW files with rm ~/uml/*
    • /etc/fstab can be modified to prevent this behavior
    • you may also be able to use tune2fs -c 0 -i 0 [root_fs_file] to turn off these checks
    • You may need to "boot up" a single UML instance without a COW file if there is an error on the root filesystem that fsck needs to fix
      • e.g. run /a/uml/debian31/linux mem=32m con1=pts umid=test ubda=/a/uml/debian31/Debian-3.1-x86-root_fs manually from a shell prompt, login as root, then halt the machine to exit cleanly
      • make sure that the root filesystem /a/uml/debian31/Debian-3.1-x86-root_fs is writeable by the user you run this command as (or else fsck will not be able to write changes and may get stuck in a continuous rebooting cycle)

  • /tmp fills up with a lot of directories
    • these are from the UML networking (uml_switch_socket), each directory contains a special FIFO file
    • e.g. /usr/bin/uml_switch -unix /tmp/tmpXoPNC1/uml_switch_socket -hub
    • when VNE/UML is not running, these files can safely be removed, rm -rf /tmp/tmp* (be careful)

  • tshark does not seem to work correctly if only 32MB of RAM is given to the UML instance
    • give the UML instance 128MB and it seems to work OK

  • UML networking does not seem to work correctly under Debian 4.0 ("etch")
    • major difference is the kernel — Debian 3.1 systems used 2.4.27 whereas Debian 4.0 systems use 2.6.18
    • don't try to use a Debian 4.0 machine as the UML host unless you want to debug this problem, stick with Debian 3.1
    • UPDATE: this actually appears to be related to the tcsetattr() problem described above

  • locale is not being set properly in the current root filesystem
    • consider adding /etc/default/locale file with LANG=C (??)

  • NARB/RCE CLI crashes when pressing backspace key
    • this is because the terminal settings are not being set correctly in UML for some reason
    • e.g. run stty -a to see rows 0; columns 0; in the output
    • then run tset -s to reset the terminal (to Linux)
    • now re-run stty -a to see rows 24; columns 80; in the output
    • if you start the DRAGON NARB/RCE in this environment, it will not crash when you press backspace

Future Work

  • add some checking to increase chances of success
    • does the directory ~/uml/ exist? if not, create it..
    • check if various files exist: screen, xterm, UML kernel, root filesystem, throw an error if they do not

  • data plane simulation
    • currently, only control plane is implemented
    • implementing data plane would allow a user to "ping" across a provisioned LSP, for example
    • use brctl/vconfig in the UML instance to setup the VLAN cross-connect
    • how should VLSR interact with this "software switch" ?
      • possible implementations include:
        • new SwitchCtrl class on VLSR that runs the appropriate brctl/vconfig commands
        • using NET-SNMP's snmpd with the pass_persist option to create an RFC2674-compliant switch
      • Update: This 'software switch' functionality is now supported in the VLSR but has not been tested yet within the UML environment.

  • make quit command in VNE text interface work correctly (shut down UML instances cleanly)

Related Work

Questions? Comments? Problems?

Please let us know if you have any questions, comments or problems, so that we can make this page better. Thanks!

-- ChrisTracy - 12 Feb 2007

End of topic
Skip to action links | Back to top

I Attachment sort Action Size Date Who Comment
uml-kernel-config manage 11.4 K 12 Feb 2007 - 16:13 ChrisTracy custom UML kernel .config file
linux-2.6.18.3.bz2 manage 941.4 K 12 Feb 2007 - 16:31 ChrisTracy linux 2.6.18.3 pre-compiled UML kernel (no debugging)
linux-2.6.18.3.debug.bz2 manage 11702.9 K 12 Feb 2007 - 16:38 ChrisTracy linux 2.6.18.3 pre-compiled UML kernel (debugging enabled)
UML_screenshot_basic_2007_02_13.png manage 206.4 K 13 Feb 2007 - 10:31 ChrisTracy UML screenshot (using the config.xml example from the VNE package)
VNE_overview.png manage 44.6 K 13 Feb 2007 - 11:41 ChrisTracy VNE overview graphic
DFN_UML.pdf manage 1230.0 K 14 Jun 2007 - 22:40 ChrisTracy paper on Advanced Network Simulation under User-Mode Linux, a good reference for this topic
anp-jvdh-gjv.pdf manage 269.5 K 15 Jun 2007 - 01:24 ChrisTracy Virtual environments for networking experiments
linux-2.6.18.3.with.hostfs.bz2 manage 1170.0 K 15 Jun 2007 - 03:07 ChrisTracy linux 2.6.18.3 pre-compiled UML kernel (no debugging, with HOSTFS support)
VNE-dragon-snapshot.2007Jun15.tar.gz manage 45.7 K 15 Jun 2007 - 03:21 ChrisTracy Virtual Network Experiments (VNE) python package — snapshot as of 2007 Jun 15
uml_linux-2.6.20 manage 6554.1 K 11 Jul 2007 - 12:14 ChrisTracy linux 2.6.20 pre-compiled UML kernel (hostfs with no crashing problems)
uml_utilities_20040406.tar.bz2 manage 58.8 K 23 Jul 2007 - 11:53 ChrisTracy locally cached copy of uml_utilities source
uml_utilities_20070815.tar.bz2 manage 49.0 K 27 Mar 2008 - 15:48 ChrisTracy locally cached copy of uml_utilities source
uml_utilities_20060323.tar.bz2 manage 44.6 K 27 Mar 2008 - 15:49 ChrisTracy locally cached copy of uml_utilities source

You are here: DRAGON > UserModeLinux

to top

Copyright © 1999-2008.
The information contained in these pages is the property of the Mid-Atlantic Crossroads (MAX).
If you have questions or comments, please contact MAX Administration