CORD : InstaCORD on CloudLab

One of the issues that the CORD team has faced is long build times for CORD-in-a-Box.  The current recipe is to download and run cord-in-a-box.sh with the following recipe:

$ curl -o ~/cord-in-a-box.sh https://raw.githubusercontent.com/opencord/cord/master/scripts/cord-in-a-box.sh
$ bash ~/cord-in-a-box.sh -t

Build times are typically > 2 hours, a significant penalty for development.

The preferred way to fix this would be to have a CloudLab profile for CORD-in-a-Box.  Unfortunately, cord-in-a-box -t builds over 70 GB of virtual machine images, too large to store in a CloudLab or to mount on  the main disk on a CloudLab machine.  Cord-in-a-box  solves this problem by loading the images into /mnt/extra/libvirt_images, and linking this to /var/lib/libvirt/images to /mnt/extra/libvirt_images.  Unfortunately, external disks aren't snapshotted as part of an image (and in fact if external disks are mounted when a node is snapshotted, the snapshotted image won't load – hangs on waiting for the external disk), so, to date, we haven't been able to build a Cord-in-a-box profile.  This problem has now been solved, by putting the built images into a couple of  tarballs in the /proj/cord-testdrive-PG0/cord-in-a-box folder  and creating custom images and scripts to automatically load the images from the tarballs and bring up the CORD VMs.  We call this project InstaCORD.  It is still not instantaneous; untarring a 35GB tarball takes on the order of 15 minutes.  

Using An InstaCORD image

All InstaCORD images must be used as user cord.

The 3.0 Dangerous Addition release build is in CIAB3.0.  Other nightly builds will be in Profiles of the form  CiaBBuildYYYY-mm-dd, which will be the profile for the Cloud-in-a-Box build on date dd/mm/YYYY. Typically, there will only be one or two such profiles; each set of tarballs take up 40 GB of disk in the /proj/cord-testdrive-PG0 partition, and there's only 100 GB there.  For this reason, before you swap in an experiment, make sure the associated tarballs are available (they should be, but deletions of tarballs without images can happen).  Nightly  tarballs will be in  /proj/cord-testdrive-PG0/cord-in-a-box/YYYY.mm.dd.  The Dangerous Addition release tarballs are in  /proj/cord-testdrive-PG0/cord-in-a-box/ciab3.0. Once the experiment is swapped in, log into node as yourself and:

$ sudo usermod -aG root cord

$ sudo usermod -aG libvirtd cord

This is required so that user cord can sudo without password, required for the restart script. Once that has succeeded

$ su cord

$ cd

$ source ./restart.sh

After about 15 minutes, the task will have completed.  When the script finishes, you should be in directory ~cord/cord/build as user cord.  In this directory

        $ vagrant status

Should show VMs corddev, prod, leaf-1, leaf-2, spine-1, and compute-node-1 as up

You can then enter the VMs and fool around, using the Cord-in-a-Box quickstart.md file as a guide.

Building an InstaCORD Image (experienced developers only)

Before you begin: check to make sure there is at least 40GB free on /proj/cord-testdrive-PG0.  If there isn't, and you know what you're doing, you can create some by deleting old tarballs and their associated images.

Then: swap in an experiment with CORDBuildMinimal.  Once again, user cord must be able to su, so

$ sudo usermod -aG root cord

grep for cord in /etc/group and make sure it's in root.  Then

$ su cord

$ ~/build.sh

This will run for about 2 hours, almost all of it executing cord-in-a-box.sh -t.   Once it is done, check for correctness:

1. tarballs created?  $ ls  /proj/cord-testdrive-PG0/cord-in-a-box/YYYY.mm.dd.  You should see cord-home.tgz and libvirt.tgz, both freshly created
2. home directory emptied?  ls -la ~ should show three .sh files and a .ssh directory
3. /mnt/extra dismounted?  $ mount | grep /mnt/extra and $ grep mnt /etc/fstab should both turn up nothing
4. libvirt stopped?  $ sudo service --status-all should show a - beside libvirt-bin
5. restart,sh successfully written?  $ grep imageDir ~/restart.sh should show /proj/cord-testdrive-PG0/cord-in-a-box/YYYY.mm.dd

If all of these tests pass, your image probably works.  Go to the experiment main page, clone, and pick the name CiaBBuildYYYY.mm.dd for the profile.  Enter "CORD-in-a-Box Build of mm/dd/YYYY" in the description, and in the instructions enter:

$ sudo usermod -aG root cord

$ su cord

$ cd

$ source ./restart.sh

Don't forget to check the "Did you add users or groups?" box.  Failing to do this will not capture system group libvirtd, which your image will need.