CORD : Implementation of Synchronizers

This document describes the implementation of synchronizers, including the synchronizer core that all synchronizers share, and the implementation of service synchronizers. For an overview of the principles involved in the Synchronizer and its design, refer to Design of Synchronizers. 

There are three types of synchronizers: Work-based Synchronizers, Event-based Synchronizers, and Hybrid Synchronizers (the last of which subsume the functionalities of the first two). Work-based synchronizers are somewhat cumbersome to implement, but offer strong robustness guarantees such as causal consistency, retries in the face of failure, model-dependency analysis and concurrent scheduling of synchronization modules. Event-based synchronizers are simpler to implement, but lack the aforementioned guarantees. 

Difference between Work-based and Event-based Synchronizers

Implementing an Event-based Synchronizer

Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service.

An Event-based Synchronizer is a collection of "Watcher" modules. Each Watcher module listens for ("watches") events pertaining to a particular model. The Synchronizer developer must provide the set of these modules. The steps for assembling a synchronizer once these modules have been implemented, are as follows:

1. Run the gen_watcher.py script: gen_watcher.py <name of your app>
2. Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to true. FIXME
3. Install python-redis by running "pip install redis" in your Synchronizer container FIXME
4. Link the redis container that comes packaged with xos with your Synchronizer container as "redis'" FIXME
5. Drop your watcher modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps
6. Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh

Watcher Module API

The main body of a watcher module is the function handle_watched_object, which responds to operations on objects that the module synchronizes. If the module responds to multiple object types, then it must determine the type of object, and proceed to process it accordingly.

def handle_watched_object(self, o):
    if (type(o) is Slice):
        self.handle_changed_slice(o)
    elif (type(o) is Node):
        self.handle_changed_node(o)

Linking the Watcher into the Synchronizer

There are two ways of linking in a Watcher. Using them both does not hurt. The first method is complex but robust, and involves making the declaration in the data model, by ensuring that the model that your synchronizer would like to watch is linked to the model that it actuates. For instance, if your synchronizer actuates a service model called Fabric, which links the Instance model, then you would ensure that Instance is a dependency of Fabric by making the following annotation in the Fabric model:

class Fabric(Service):
    ...
    ...
    xos_links = [ModelLink(Instance,via='instance',into='ip')]

There can be several ModelLink specifications in a single xos_links declaration, each encapsulating the referenced model, the field in the current model that links to it, and the destination field in which the watcher is interested. If into is omitted, then the watcher is notified of all changes in the linked model, irrespective of the fields that change. 

The above change needs to be backed up with an instruction to the synchronizer that the watcher is interested in being notified of changes to its dependencies. This is done through a watch_degree annotation.

class SyncFabricService(SyncStep):
    watch_degree=1

By default, watch_degree is 0, meaning that the Synchronizer watches nothing. When watch degree is 1, it watches 1 level of dependencies removed, and so on. If the watch_degree in the above code were 2, then this module would also get notified of changes in dependencies of the Instance model.

The second way of linking in a watcher is to hardcode the watched model directly in the synchronizer:

class SyncFabricService(SyncStep):
    watched = [ModelLink(Instance,via='instance',into='ip')]

Linking the Watcher into the Synchronizer

• Set the observer_enable_watchers option to true in your xos synchronizer config file
• Add a link between your synchronizer container and the redis container by including the following lines in the definition of your synchronizer's docker-compose file. You may need to adapt these to the name of the project used (e.g. cordpod)
  • external_links: 
    
    • xos_redis:redis
      
      
• Ensure that there is a similar link between your XOS UI container and the redis container

In addition to the above development tasks, you also need to make the following changes to your configuration to activate watchers.

Implementing a Work-based Synchronizer

Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service.

A work-based Synchronizer is a collection of "Actuator" modules. Each Actuator module is invoked when a model is found to be outdated relative to its last synchronization. An actuator module can be self-contained and written entirely in Python, or it can be broken into a "dispatcher" and "payload", with the dispatcher implemented in Python and the payload implemented using Ansible. The Synchronizer core has built-in support for the dispatch of Ansible modules and helps extract parameters from the synchronized model and translate them into the parameters required by the corresponding Ansible script. It also tracks an hierarchically structured list of such ansible scripts on the filesystem, for operators to use to inspect and debug a system. The procedure for building a work-based synchronizer is as follows:

1. Run the gen_workbased.py script. gen_workbased <app name>.
2. Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to False. FIXME
3. Drop your actuator modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps
4. Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh

Actuator Module API

Implementing a Step with Ansible

To implement a step using Ansible, a developer must provide two things: an Ansible recipe, and a get_extra_attributes method, which maps attributes of the object into a dictionary that configures that Ansible recipe. The Ansible recipe comes in two parts, an inner payload and a wrapper that delivers that payload to the VMs associated with the service. The wrapper itself comes in two parts. A part that sets up the preliminaries:

---
 
- hosts: "{{ instance_name }}"
  connection: ssh
  user: ubuntu
  sudo: yes
  gather_facts: no
  vars:
    - package_location: "{{ package_location }}"
    - server_port: "{{ server_port }}"

The template variables package_location and server_port come out of the Python part of the Synchronizer implementation (discussed shortly).

The outer wrapper then includes a set of Ansible roles that perform the required actions:

roles:
  - download_packages
  - configure_packages
  - start_server

The "payload" of the Ansible recipe contains an implementation of the roles, in this case, download_packages, configure_packages and start_server.

The concrete values of parameters required by the Ansible payload are provided in the implementation of the "get_extra_attributes" method in the Python part of the Synchronizer. This method receives an object from the data model and is charged with the task of converting the properties of that object into the set of properties required by the Ansible recipe, which are returned as a Python dictionary.

def get_extra_attributes(self, o):
        fields = {}
        fields['package_location'] = o.package_location
        fields['server_port'] = o.server_port
        return fields

Implementing a Step without Ansible

To implement a step without using Ansible, a developer need only implement the sync_record and delete_record methods, which get called for every pending object. These methods interact directly with the underlying substrate. 

Managing Dependencies

If your data models have dependencies between them, so that for one to be synchronized, another must already have been synchronized, then you can define such dependencies in your data model. The Synchronizer automatically picks up such dependencies and ensures that the steps corresponding to the models in questions are executed in a valid order. It also ensures that any errors that arise propagate from the affected objects to its dependents, and that the dependents are held up until the errors have been resolved and the dependencies have been successfully synchronized.

In the absence of failures, the Synchronizer tries to execute your synchronization steps concurrently to whatever extent this is possible while still honoring dependencies.

    <in the definition of your model>
    xos_links = [ModelLink(dest=MyServiceUser,via='user'),ModelLink(dest=MyServiceDevice,via='device')] 

In the above example, the xos_links field declares two dependencies. The name "xos_links" is key, and so the field should be named as such. The dependencies are contained in a list of type ModelLink, each of which defines a type of object (a model) and an "accessor" field via which a related object of that type can be accessed.

Handling Errors

To fault synchronization, you can raise an exception in any of the methods of your step that are automatically called by the synchronizer core. These include fetch_pending, sync_record and delete_record. The outcome of such exceptions has multiple parts:

1. The synchronization of the present object is deferred.
2. The synchronization of dependent objects is deferred, if those objects are accessible via the current object (see the 'via' field)
3. A string representation of your exception is propagated into a scratchpad in your model, which in turn appears in your UI. When you click the object in question, in the UI, you should see the error message.
4. The synchronization state of your object, and of dependent objects changes to "Error" and a red icon appears next to it.
5. If the object repeatedly fails to synchronize, then its synchronization interval is increased exponentially.

Sometimes, you may encounter a temporary error, which you think may be resolved shortly, by the time the Synchronizer runs again. In these cases, you can raise a "DeferredException." This error type differs from a general exception in two ways:

1. It does not put your object in error state
2. It disabled exponential backoff - the Synchronizer tries to synchronize your object every single time.

Synchronizer Configuration Options