CORD : M-CORD Developer Environment for CORD 2.0

Part 1: Step up the development environment

There are several ways to build the development environment for M-CORD.

This document just introduce one way. I followed: with CloudLab.

Assume you already have an CloudLab account. Step1 - 3  are creating a ubuntu server on CloudLab.

Step 0: Inside your account, click "Actions" button, then choose "Start Experiment"

Step 1: Select the profile "OnePC-Ubuntu14.04.4", and click Next.

Step 2: Give a name, and in the “Cluster” dropdown select “Cloudlab Clemson”

Step 3: Click next until finish. Click "List View" you can see the ssh command line for how to login your server.


Look at the Expire date, you may need to extend it. If you extend less than 7 days, it will automatically get approval. But still can not extend for ever, after extending twice, you will get 14 days, then the administrator will not let you continue.

This is the limitation on CloudLab, if you have your own server for development, then you do not have such issue.


Step 4 - 9 are showing how to run M-CORD.

Step 4: Use the command line in Step 3, to login your server. Follow directions in CORD-in-a-Box Quick Start Guide.


sudo apt-get update
sudo apt-get install curl

curl -o ~/
bash ~/ -t | tee ~/install.out

Until it finish.

NOTE:  The cord-in-a-box script takes a long time (at least three hours) to run.  It is highly recommended that you run it using a terminal multiplexer.  


Step 5: Login XOS VM, you will see there folders.

pingping@node:~$ cd opencord/build/
pingping@node:~/opencord/build$ ssh prod
vagrant@prod:~$ ls  keystone_juju_ca_cert.crt  node_key  onos-cord  onos-fabric  service-profile  xos  xos_libraries  xos_services

service-profile folder contains the M-CORD profiles.

xos folder contains the basic xos functions.

xos_services contains all the services, such as vBBU, vSGW, vMME, vPGWC...

Step 5: Reset the CORD environment.

vagrant@prod:~$ cd ~/service-profile/cord-pod
vagrant@prod:~/service-profile/cord-pod$ make cleanup


Step 7: Prepare the M-CORD environment.

vagrant@prod:~$ cd service-profile/mcord/
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/ .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/id_rsa* .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/node_key .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/nodes.yaml .


Step 8: Cleanup CORD-POD environment.

vagrant@prod:~/service-profile/cord-pod$ make stop
vagrant@prod:~/service-profile/cord-pod$ make cleanup

If you changed xos, you also need to run:

vagrant@prod:~/service-profile/mcord$ make local_containers


Step 9: Start M-CORD environment.

Go to the services folder, see what is inside.

vagrant@prod:~/service-profile/mcord$ cd ~/xos_services
vagrant@prod:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  olt  onos-service  openstack  vrouter  vsg  vtn  vtr

The M-CORD services are not here. Lets pull M-CORD services from Gerrit, the project list, you can see vBBU, vPGWC, vSGW, vSGW.

vagrant@prod:~/service-profile/mcord$ make pull_services
vagrant@prod:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  metronet-local  olt  onos-service  openstack  vBBU  vMME  vPGWC  vrouter  vsg  vSGW  vtn  vtr

Right now you can see the vBBU, vPGWC, vSGW,  vSGW now. 


Step 10: Run M-CORD. 

vagrant@prod:~/service-profile/mcord$ make xos
vagrant@prod:~/service-profile/mcord$ make vtn
(if you using fabric )
vagrant@prod:~/service-profile/mcord$ make fabric
vagrant@prod:~/service-profile/mcord$ make mcord


Step 11: Check whether the configuration was loaded into the system

After "make vtn" in step 9, the network config should be loaded into ONOS. Below is how to check, with password "karaf".

pingping@node:~$ ssh -p 8101 karaf@onos-cord
Password authentication
Welcome to Open Network Operating System (ONOS)!
     ____  _  ______  ____     
    / __ \/ |/ / __ \/ __/   
   / /_/ /    / /_/ /\ \     
Mailing lists:     

Come help out! Find out how at: 

Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

onos> netcfg
  "apps" : {
    "org.opencord.vtn" : {
      "cordvtn" : {
        "localManagementIp" : "",
        "nodes" : [ {
          "bridgeId" : "of:0000000000000001",
          "dataPlaneIntf" : "fabric",
          "dataPlaneIp" : "",
          "hostManagementIp" : "",
          "hostname" : "nova-compute-1"
        } ],
        "openstack" : {
          "endpoint" : "https://keystone.cord.lab:5000/v2.0",
          "password" : "VeryLongKeystoneAdminPassword",
          "tenant" : "admin",
          "user" : "admin"
        "ovsdbPort" : "6641",
        "privateGatewayMac" : "00:00:00:00:00:01",
        "publicGateways" : [ ],
        "ssh" : {
          "sshKeyFile" : "/root/node_key",
          "sshPort" : "22",
          "sshUser" : "root"
        "xos" : {
          "endpoint" : "http://xos/",
          "password" : "letmein",
          "user" : ""
  "links" : { },
  "devices" : { },
  "hosts" : { },
  "ports" : { }


If the there is no data after "onos> netcfg", which means during  step "make vtn", the configuration was not loaded into the system. This is an known issue, the XOS team is fixing it. After fixing, we do not need to do the (1), (2), (3) below.

Right now, we can do the following to solve it:

(1) Run the follow command line on your local computer. XOS portal for M-CORD uses port 8080

ssh -L 8888:xos:8080

(2) open your bower with http://localhost:8888, login with user name, Password: letmein.  You can also go to http://<ip_address_of_server>:8888.  

(3) click "Service" --> "ONOS-CORD"

After enter ONOS_CORD, then click the "Administraion" tab, then click "ONOS APPs"

After enter "ONOS APPs", then click "VTN_ONOS_app"

After enter "VTN_ONOS_app", then click "Save" button.

Then log in onos, and check the config again.


Part 2: Verification by creating a new service instance/vm.

1. There should be 9 containers at this moment. It may change in the future when we add more services.


 2. Run the follow command line on your local computer. XOS portal for M-CORD uses 8080.

ssh -L 8888:xos:8080


3. Right now, you can open your bower with http://localhost:8888, and you can see the M-CORD relative services now.

The user name is:, Password: letmein

4. Boot an instance.

Click "Slices", then click "mysite_vbbu_slice1"

Then Click "Instances" tab, then click "Add instance"

Note: must choose nova-compute-1 and trusty-server-multi-nic. Other nodes and iamges are not available on CloudLab, will be only available on M-CORD rack.

Then click "Save"

Check the Instance again, if you see the green , then it means you succeed.

Part 3: How to develop a M-CORD service inside XOS?


XOS team put the latest tutorial here:
    Note: The video if for an old version of XOS, but you still can learn some concept and ideas from it.
There's also a page that describes the on-boarding internals a bit more:


Also, on, we can discuss any issue you meet in the "#xos" channel.

Part 4: How to update the entire environment for M-CORD


Normally we need to use the "git pull" to update the following projects:











ubuntu@xos-1:~/service-profile$ git branch
* master
ubuntu@xos-1:~/service-profile$ git pull
ubuntu@xos-1:~/service-profile$ cd ../xos
ubuntu@xos-1:~/xos$ git branch
* master
ubuntu@xos-1:~/xos$ git pull
ubuntu@xos-1:~/xos$ cd ../xos_services/
ubuntu@xos-1:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  monitoring  olt  onos-service  openstack  vBBU  vPGWC  vrouter  vsg  vtn  vtr
ubuntu@xos-1:~/xos_services$ cd vBBU/
ubuntu@xos-1:~/xos_services/vBBU$ git branch
* master
ubuntu@xos-1:~/xos_services/vBBU$ git pull
ubuntu@xos-1:~/xos_services$ cd vPGWC/
ubuntu@xos-1:~/xos_services/vPGWC$ git branch
* master
ubuntu@xos-1:~/xos_services/vPGWC$ git pull

Same to vBBU and vPGW, you should git pull the "vtn", "onos-service", "openstack", "monitoring".


Then in "ubuntu@xos-1:~/service-profile$" repeat Step 7, and Step 9 to rebuild.

Part 5: How to restart ONOS

ONOS is running in a container inside prod vm.

From you CloudLab server, log in the prod vm.

How to stop:

pingping@node:~$ ssh prod
vagrant@prod:~$ cd onos-cord/
vagrant@prod:~/onos-cord$ ls
docker-compose.yml  Dockerfile  node_key  onos-service  xos-certs.crt
vagrant@prod:~/onos-cord$ docker-compose kill xos-onos

How to start:

vagrant@prod:~/onos-cord$ docker-compose up -d

How to verify is ONOS is running


spirent@cord4:~/onos-cord$ docker ps | grep onos
b5be3f4c13d9 xosproject/xos-synchronizer-onos "bash -c 'sleep 12..." 2 days ago Up 2 days mcord_xos_synchronizer_onos_1
f2b95a019ea7 xos/onos "./bin/onos-service" 13 days ago Up 50 seconds>6653/tcp,>8101/tcp,>8181/tcp,>9876/tcp onoscord_xos-onos_1


How to verify is ONOS is running


mcord@cord4:~/onos-cord$ docker ps | grep onos
b5be3f4c13d9 xosproject/xos-synchronizer-onos "bash -c 'sleep 12..." 2 days ago Up 2 days mcord_xos_synchronizer_onos_1
f2b95a019ea7 xos/onos "./bin/onos-service" 13 days ago Up 50 seconds>6653/tcp,>8101/tcp,>8181/tcp,>9876/tcp onoscord_xos-onos_1

if you can see "onoscord_xos-onos_1", then it means onos-cord container is running.