Created by Srikanth Vavilapalli, last modified on Mar 14, 2017

 

This page describes the instructions to bring up monitoring service components using new build process.

 

Monitoring Service is not very stable in the latest master branches because of some issues in synchronizer execution of playbooks. You might notice failures at different stages of monitoring service deployment process. Refer to Troubleshooting section

 

For convenience, add the below alias to your bashrc file and source it:

alias xos-teardown="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml teardown-playbook.yml"
alias xos-cleanup-images="docker rmi xosproject/xos-ui xosproject/xos"
alias xos-cleanup-openstack="source ~/admin-openrc.sh; /opt/cord/orchestration/service-profile/cord-pod/cleanup.sh"
alias xos-deploy="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml deploy-xos-playbook.yml"
alias compute-node-refresh="pushd /opt/cord/build/platform-install; ansible-playbook -i /etc/maas/ansible/pod-inventory --extra-vars=@/opt/cord/build/genconfig/config.yml compute-node-refresh-playbook.yml"
 
 
alias deploy-monitoringservice="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml onboard-monitoringservice-playbook.yml"
alias instantiate-monitoringservice="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml instantiate-monitoringservice-playbook.yml"
alias generate-install-infra-agents-config="ansible-playbook -i /etc/maas/ansible/pod-inventory --extra-vars=@/opt/cord/build/genconfig/config.yml monitoringagent-infra-install-config.yml"
alias install-infra-agents="ansible-playbook -i /opt/cord_profile/monitoringagent_install.hosts monitoringagent-infra-install.yml"
alias generate-enable-infra-monitoring-config="ansible-playbook -i /etc/maas/ansible/pod-inventory --extra-vars=@/opt/cord/build/genconfig/config.yml monitoringservice-enable-inframonitoring-config.yml"
alias enable-infra-monitoring="ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-enable-inframonitoring.yml"
alias enable-vsg-monitoring="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-enable-vsgmonitoring.yml"
alias enable-exampleservice-monitoring="pushd /opt/cord/build/platform-install; ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-enable-exampleservicemonitoring.yml"
alias test-monitoringservice-initial="ansible-playbook -i inventory/rcord  --extra-vars @../genconfig/config.yml monitoringservice-test-initial.yaml"
alias test-monitoringservice-inframetrics="ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-test-inframetrics.yaml"
alias test-monitoringservice-vsgmetrics="ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-test-vsgmetrics.yaml"
alias test-monitoringservice-exampleservicemetrics="ansible-playbook -i inventory/rcord --extra-vars @../genconfig/config.yml monitoringservice-test-exampleservicemetrics.yaml"

alias cord-monitoringservice="echo \"*******NOTE:Ensure monitoring service synchronizer is completely up and running********\"; instantiate-monitoringservice; test-monitoringservice-initial"
alias install-inframonitoring-agent="generate-install-infra-agents-config; install-infra-agents"
alias enable-inframonitoring="generate-enable-infra-monitoring-config; enable-infra-monitoring; test-monitoringservice-inframetrics"
alias enable-vsgmonitoring="enable-vsg-monitoring; test-monitoringservice-vsgmetrics"
alias enable-exampleservicemonitoring="enable-exampleservice-monitoring; test-monitoringservice-exampleservicemetrics"

 

Install CORD POD on a server (cord-in-a-box) using the latest install procedures as described in Configuring CORD

  • Ensure the CORD portal is up and all the mandatory services are up and running
  • By default, POD install on-boards monitoring service synchronizer 

Ensure monitoring service synchronizer is operational by checking the logs from the respective synchronizer docker container

Install Monitoring Service & Channel 

vagrant@prod:~$ cd /opt/cord/build/platform-install
vagrant@prod:/opt/cord/build/platform-install$ source ~/.bashrc
vagrant@prod:/opt/cord/build/platform-install$ cord-monitoringservice

`cord-monitoringservice` alias performs the following functionalities:

  • Instantiation of Monitoring service
  • Creation of Monitoring channel (for user xosadmin@opencord.org)
  • Loading Monitoring dashboard that retrieves the data using Monitoring channel
  • Run basic test suite that verifies the status of Monitoring service and channel and also ensures there are no telemetry data available initially

Enable Monitoring of Infrastructure Services (OpenStack and ONOS)

vagrant@prod:/opt/cord/build/platform-install$ install-inframonitoring-agent
vagrant@prod:/opt/cord/build/platform-install$ enable-inframonitoring
  • The first alias `install-inframonitoring-agents` will deploy the monitoring agents in all the servers (head nodes and compute nodes). These monitoring agents provide an interface to control monitoring of the various infrastructure services (OpenStack and ONOS)
  • The second alias `enable-inframonitoring` will invokes the APIs exposed by monitoring agents (deployed in previous command) and turn ON the monitoring for OpenStack and ONOS services. At the end, it runs a test suite that validates if the telemetry data is collected from these infrastructure services.

 

Enable Monitoring of R-CORD vSG Services

vagrant@prod:/opt/cord/build/platform-install$ enable-vsgmonitoring

 

Enable Monitoring of Example Services

vagrant@prod:/opt/cord/build/platform-install$ enable-exampleservicemonitoring

 

 

Troubleshooting

  • 'cord-monitoringservice' command does not bring up Monitoring service:
    • Two reasons:
      • ONOS-CORD connectivity issues:
        • Off-late, due to some 'StorgaeTimeout' exception issues in ONOS, the VMs created for a given service are not reachable by XOS synchronizer and hence the service configuration will never complete in this case.
        • Refer to instructions at "Restart ONOS-CORD" to restart the ONOS and verify the connectivity to the VMs are working properly
      • Longer ansible playbook execution times
        • Sometimes, we have noticed that synchronizer ansible playbook execution was taking more than expected time, due to which, monitoring service test cases would flag failure.
        • In such case, re-run the test case command 'test-monitoringservice-initial' after giving a small pause to see if it report success