CORD : M-CORD Developer Environment for CORD 2.0

Part 1: Step up the development environment

There are several ways to build the development environment for M-CORD.

This document just introduce one way. I followed:  https://github.com/opencord/cord/blob/cord-2.0/docs/quickstart.md with CloudLab.

Assume you already have an CloudLab account. Step1 - 3  are creating a ubuntu server on CloudLab.

Step 0: Inside your account, click "Actions" button, then choose "Start Experiment"

Step 1: Select the profile "OnePC-Ubuntu14.04.4", and click Next.

Step 2: Give a name, and in the “Cluster” dropdown select “Cloudlab Clemson”

Step 3: Click next until finish. Click "List View" you can see the ssh command line for how to login your server.

Step 4 - 9 are showing how to run M-CORD.

Step 4: Use the command line in Step 3, to login your server. Follow directions in CORD-in-a-Box Quick Start Guide.

Run:

sudo apt-get update
sudo apt-get install curl

curl -o ~/cord-in-a-box.sh https://raw.githubusercontent.com/opencord/cord/cord-2.0/scripts/cord-in-a-box.sh
bash ~/cord-in-a-box.sh -t | tee ~/install.out

Until it finish.

NOTE:  The cord-in-a-box script takes a long time (at least three hours) to run.  It is highly recommended that you run it using a terminal multiplexer.  

Step 5: Login XOS VM, you will see there folders.

pingping@node:~$ cd opencord/build/
pingping@node:~/opencord/build$ ssh prod
vagrant@prod:~$ ls
admin-openrc.sh  keystone_juju_ca_cert.crt  node_key  onos-cord  onos-fabric  service-profile  xos  xos_libraries  xos_services

service-profile folder contains the M-CORD profiles.

xos folder contains the basic xos functions.

xos_services contains all the services, such as vBBU, vSGW, vMME, vPGWC...

Step 5: Reset the CORD environment.

vagrant@prod:~$ cd ~/service-profile/cord-pod
vagrant@prod:~/service-profile/cord-pod$ make cleanup

Step 7: Prepare the M-CORD environment.

vagrant@prod:~$ cd service-profile/mcord/
vagrant@prod:~/service-profile/mcord$ 
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/admin-openrc.sh .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/id_rsa* .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/node_key .
vagrant@prod:~/service-profile/mcord$ cp ../cord-pod/nodes.yaml .

Step 8: Cleanup CORD-POD environment.

vagrant@prod:~/service-profile/cord-pod$ make stop
vagrant@prod:~/service-profile/cord-pod$ make cleanup

If you changed xos, you also need to run:

vagrant@prod:~/service-profile/mcord$ make local_containers

Step 9: Start M-CORD environment.

Go to the services folder, see what is inside.

vagrant@prod:~/service-profile/mcord$ cd ~/xos_services
vagrant@prod:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  olt  onos-service  openstack  vrouter  vsg  vtn  vtr

The M-CORD services are not here. Lets pull M-CORD services from Gerrit, https://gerrit.opencord.org/#/admin/projects/. On the project list, you can see vBBU, vPGWC, vSGW, vSGW.

vagrant@prod:~/service-profile/mcord$ make pull_services
vagrant@prod:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  metronet-local  olt  onos-service  openstack  vBBU  vMME  vPGWC  vrouter  vsg  vSGW  vtn  vtr

Right now you can see the vBBU, vPGWC, vSGW,  vSGW now. 

Step 10: Run M-CORD. 

vagrant@prod:~/service-profile/mcord$ make xos
vagrant@prod:~/service-profile/mcord$ make vtn
(if you using fabric )
vagrant@prod:~/service-profile/mcord$ make fabric
vagrant@prod:~/service-profile/mcord$ make mcord

Step 11: Check whether the configuration was loaded into the system

After "make vtn" in step 9, the network config should be loaded into ONOS. Below is how to check, with password "karaf".

pingping@node:~$ ssh -p 8101 karaf@onos-cord
Password authentication
Password: 
Welcome to Open Network Operating System (ONOS)!
     ____  _  ______  ____     
    / __ \/ |/ / __ \/ __/   
   / /_/ /    / /_/ /\ \     
   \____/_/|_/\____/___/     
                               
Documentation: wiki.onosproject.org      
Tutorials:     tutorials.onosproject.org 
Mailing lists: lists.onosproject.org     

Come help out! Find out how at: contribute.onosproject.org 

Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

onos> netcfg
{
  "apps" : {
    "org.opencord.vtn" : {
      "cordvtn" : {
        "localManagementIp" : "10.102.83.2/24",
        "nodes" : [ {
          "bridgeId" : "of:0000000000000001",
          "dataPlaneIntf" : "fabric",
          "dataPlaneIp" : "10.168.0.253/24",
          "hostManagementIp" : "192.168.122.140/24",
          "hostname" : "nova-compute-1"
        } ],
        "openstack" : {
          "endpoint" : "https://keystone.cord.lab:5000/v2.0",
          "password" : "VeryLongKeystoneAdminPassword",
          "tenant" : "admin",
          "user" : "admin"
        },
        "ovsdbPort" : "6641",
        "privateGatewayMac" : "00:00:00:00:00:01",
        "publicGateways" : [ ],
        "ssh" : {
          "sshKeyFile" : "/root/node_key",
          "sshPort" : "22",
          "sshUser" : "root"
        },
        "xos" : {
          "endpoint" : "http://xos/",
          "password" : "letmein",
          "user" : "padmin@vicci.org"
        }
      }
    }
  },
  "links" : { },
  "devices" : { },
  "hosts" : { },
  "ports" : { }
}

If the there is no data after "onos> netcfg", which means during  step "make vtn", the configuration was not loaded into the system. This is an known issue, the XOS team is fixing it. After fixing, we do not need to do the (1), (2), (3) below.

Right now, we can do the following to solve it:

(1) Run the follow command line on your local computer. XOS portal for M-CORD uses port 8080

ssh -L 8888:xos:8080 pingping@clnode096.clemson.cloudlab.us:

(2) open your bower with http://localhost:8888, login with user name padmin@vicci.org, Password: letmein.  You can also go to http://<ip_address_of_server>:8888.  

(3) click "Service" --> "ONOS-CORD"

After enter ONOS_CORD, then click the "Administraion" tab, then click "ONOS APPs"

After enter "ONOS APPs", then click "VTN_ONOS_app"

After enter "VTN_ONOS_app", then click "Save" button.

Then log in onos, and check the config again.

Part 2: Verification by creating a new service instance/vm.

1. There should be 9 containers at this moment. It may change in the future when we add more services.

 2. Run the follow command line on your local computer. XOS portal for M-CORD uses 8080.

ssh -L 8888:xos:8080 pingping@clnode096.clemson.cloudlab.us

3. Right now, you can open your bower with http://localhost:8888, and you can see the M-CORD relative services now.

The user name is: padmin@vicci.org, Password: letmein

4. Boot an instance.

Click "Slices", then click "mysite_vbbu_slice1"

Then Click "Instances" tab, then click "Add instance"

Note: must choose nova-compute-1 and trusty-server-multi-nic. Other nodes and iamges are not available on CloudLab, will be only available on M-CORD rack.

Then click "Save"

Check the Instance again, if you see the green , then it means you succeed.

Part 3: How to develop a M-CORD service inside XOS?

Part 4: How to update the entire environment for M-CORD

Normally we need to use the "git pull" to update the following projects:

service-profile

xos

xos_services/vBBU

xos_services/vPGWC

xos_services/vtn

xos_services/onos-service

xos_services/openstack

xos_services/monitoring

ubuntu@xos-1:~/service-profile$ git branch
* master
ubuntu@xos-1:~/service-profile$ git pull
ubuntu@xos-1:~/service-profile$ cd ../xos
ubuntu@xos-1:~/xos$ git branch
* master
ubuntu@xos-1:~/xos$ git pull
ubuntu@xos-1:~/xos$ cd ../xos_services/
ubuntu@xos-1:~/xos_services$ ls
exampleservice  fabric  hypercache  metro-net  monitoring  olt  onos-service  openstack  vBBU  vPGWC  vrouter  vsg  vtn  vtr
ubuntu@xos-1:~/xos_services$ cd vBBU/
ubuntu@xos-1:~/xos_services/vBBU$ git branch
* master
ubuntu@xos-1:~/xos_services/vBBU$ git pull
ubuntu@xos-1:~/xos_services$ cd vPGWC/
ubuntu@xos-1:~/xos_services/vPGWC$ git branch
* master
ubuntu@xos-1:~/xos_services/vPGWC$ git pull
 

Same to vBBU and vPGW, you should git pull the "vtn", "onos-service", "openstack", "monitoring".

Then in "ubuntu@xos-1:~/service-profile$" repeat Step 7, and Step 9 to rebuild.

Part 5: How to restart ONOS

ONOS is running in a container inside prod vm.

From you CloudLab server, log in the prod vm.

How to stop:

pingping@node:~$ ssh prod
 
vagrant@prod:~$ 
vagrant@prod:~$ cd onos-cord/
vagrant@prod:~/onos-cord$ ls
docker-compose.yml  Dockerfile  node_key  onos-service  xos-certs.crt
vagrant@prod:~/onos-cord$ docker-compose kill xos-onos

How to start:

vagrant@prod:~/onos-cord$ docker-compose up -d

How to verify is ONOS is running

spirent@cord4:~/onos-cord$ docker ps | grep onos
b5be3f4c13d9 xosproject/xos-synchronizer-onos "bash -c 'sleep 12..." 2 days ago Up 2 days mcord_xos_synchronizer_onos_1
f2b95a019ea7 xos/onos "./bin/onos-service" 13 days ago Up 50 seconds 0.0.0.0:6654->6653/tcp, 0.0.0.0:8102->8101/tcp, 0.0.0.0:8182->8181/tcp, 0.0.0.0:9877->9876/tcp onoscord_xos-onos_1

How to verify is ONOS is running

mcord@cord4:~/onos-cord$ docker ps | grep onos
b5be3f4c13d9 xosproject/xos-synchronizer-onos "bash -c 'sleep 12..." 2 days ago Up 2 days mcord_xos_synchronizer_onos_1
f2b95a019ea7 xos/onos "./bin/onos-service" 13 days ago Up 50 seconds 0.0.0.0:6654->6653/tcp, 0.0.0.0:8102->8101/tcp, 0.0.0.0:8182->8181/tcp, 0.0.0.0:9877->9876/tcp onoscord_xos-onos_1

if you can see "onoscord_xos-onos_1", then it means onos-cord container is running.