CORD : Dynamic On-boarding System and Service Profiles (Mysterious-Decsion)

Created by Scott Baker, last modified by Larry Peterson on May 11, 2017

Summary

This document describes the design of the CORD dynamic on-boarding system. The dynamic on-boarding system is a feature of XOS. It is responsible for adding new services to a running instance of CORD. It handles rebuilding the XOS UI container to integrate new data models and new admin pages, as well as rebuilding and managing the service's synchronizer container.

This onboarding system was implemented by the Mysterious-Decision release of CORD (cord-2.0). It has been obsoleted in Dangerous-Addition (cord-3.0) by a new build-time on-boading system.

Important Data Models

The data models used by the on-boarding process are:

  • ServiceController. Each Service should have a ServiceController object. The ServiceController object tells the name of the service, the base_url where the service's resources can be downloaded from, and holds pointers to all of the resources.
  • ServiceResource. A resource is a file or set of files that are used by the service. For example, a resource may be a models.py file that holds the models for the service, it may be a manifest that describes the service's synchronizer, or it may be a key that is important to the service. A typical service will have somewhere around a dozen resources.
  • XOS. XOS is the root model for a given deployment. It has several important parameters for XOS, such as the port number that XOS should listen on, the docker project name that should be used, names of docker base images, etc. The XOS model also has pointers to all of the ServiceController objects.

XOS Containers

XOS is deployed in a series of containers. There are three classes of containers:

  • DB
  • UI
  • Synchronizer

There is one DB container per deployment. It is used by the UI and Synchronizer containers. There are typically two UI containers: a bootstrap_ui container and a regular ui container. There are many synchronizers: an on-boarding synchronizer, an OpenStack synchronizer, and typically a synchronizer for each service.

The DB Container

The database container runs PostgreSQL. It listens on tcp port 5432. The database is mounted as a docker volume on the host at location (todo). 

Note: check pointing the db container does not necessarily back up the contents of the database, due to the volume mount.

The Bootstrap_UI container

The Bootstrap_UI container is a user interface container that is used for on-boarding services. It contains data models, REST API, TOSCA, and a fully functional user interface. The bootstrap_ui containers is populated with models for the XOS core, but it is not aware of service specific models, APIs, etc. 

The bootstrap UI container primarily serves as a place to run the TOSCA recipes that will on-board the services. The bootstrap UI container should not serve as a general purpose user interface (as it lacks service specific models) but it can serve as an emergency UI if the primary UI container is not working properly. The bootstrap_ui container often listens on a nonstandard port, such as port 81 (cord-pod configuration) or port 9998 (frontend configuration)

The Onboarding Synchronizer Container

The On-boarding Synchronizer continuously watches the ServiceController, ServiceResource, and XOS models for changes. When a change to a ServiceController or ServiceResource occurs, this causes any synchronizer container for the associated service to be rebuilt. Modifying any of the three watched models should cause the XOS UI container to be rebuilt.

The On-boarding Synchronizer uses a Docker volume to mount the host's Docker socket. This allows the On-boarding Synchronizer to build containers and deploy them using the host's Docker environment.

The method that the On-boarding Synchronizer uses to create the UI container and the Service Synchronizer containers is to use a fairly complete base Docker image, and then add the various service-specific files to that container. The set of files it adds is mostly equivalent to the set of ServiceResource objects.

The OpenStack Synchronizer Container

The OpenStack synchronizer container is automatically started for those deployments that run on top of OpenStack.

The UI Container

The UI container is maintained by the On-boarding Synchronizer. As the On-boarding Synchronizer usually takes a short while to run, the UI container will be started a short while after the On-boarding Synchronizer container. The UI container contains a full XOS UI and data model, with TOSCA and REST API for the various servers that have been on-boarded. One can think of this as UI = Bootstrap_UI + service_models + service_apis + service_admins.

The UI container generally listens on a standard HTTP port for web services, such as port 80 (cord-pod config).

Service Synchronizer Containers

Typically, each service will contain one synchronizer container that implements the service's synchronizer. The On-boarding synchronizer automatically creates and maintains these containers.

Service Profiles

A service profile is a configuration of XOS that includes a set of services. One particular service profile is the cord-pod service profile, which implements the R-CORD service graph. At this time a service profile is primarily a Makefile with some additional support files. The Makefile executes the following phases:

  1. prereqs. Install necessary software, such as Docker.
  2. dirs. Create directories that will be mounted as Docker volumes. While docker-compose will automatically create these directories if they don't exist, it will do so with root permission. Creating them from the Makefile first allows them to be created without root permission.
  3. download_xos. Downloads XOS from https://gerrit.opencord.org/p/xos.git. If XOS is already present then it won't be downloaded.
  4. download_services. Downloads XOS services from https://gerrit.opencord.org/p/<servicename>.git. If the services are already present then they won't be downloaded.
  5. bootstrap. The bootstrap phase is comprised of several sub-steps:
    1. Run docker-compose to start the db, bootstrap_ui, and onboarding_synchronizer containers. The recipe that launches these is in docker-compose-bootstrap.yaml.
    2. Execute the TOSCA recipes fixtures.yaml and mydeployment.yaml. These two recipes create various XOS models for commons roles, flavors, etc., and create the default Site, Deployment, and administrative user.
    3. Execute the TOSCA recipe xos.yaml. This recipe tells the onboarding synchronizer how to configure the UI container. For example, it tells what port the UI container should listen on, and what docker volumes should be mounted.
  6. onboarding. Onboarding is the process of loading the controllers for the service stack into XOS. This is broken into several sub-steps:
    1. Disable onboarding. When onboarding several services at once, it's convenient to pause the onboarding synchronizer and let it initialize the full service stack at once.
    2. Place public and private keys in the key_import directory.
    3. Execute each service's onboarding TOSCA recipe. These recipes typically create ServiceController and ServiceResource objects for the service.
    4. Execute the profile's synchronizers.yaml TOSCA recipe. This is an optional step. Sometimes a service profile has additional configuration parameters that need to be applied to the ServiceController objects. For example, the cord-pod service profile injects custom configuration files for the vsg and vtr synchronizers.
    5. Enable onboarding. Now that we have the whole service stack specified, it's time to turn on onboarding and let the Onboarding Synchronizer run.
    6. Wait for each service to be onboarded.
    7. Wait for the XOS UI container to be listening on its web port.
  7. Service configuration. Now that we've onboarded the ServiceControllers, it's time to run the various recipes that configure those services. Note that the TOSCA recipes in this section will usually be run against the UI container rather than the Bootstrap_UI container, as these recipes contain service-specific Tosca. The cord-pod service profile is complex and implements several different make targets to accomplish this step: podconfig, vtn, fabric, and cord.

At this point, XOS (and CORD as a whole) is running and ready for use.