CORD : Implementation of Synchronizers

This document describes the implementation of synchronizers, including the synchronizer core that all synchronizers share, and the implementation of service synchronizers. For an overview of the principles involved in the Synchronizer and its design, refer to Design of Synchronizers

There are three types of synchronizers: Work-based Synchronizers, Event-based Synchronizers, and Hybrid Synchronizers (the last of which subsume the functionalities of the first two). Work-based synchronizers are somewhat cumbersome to implement, but offer strong robustness guarantees such as causal consistency, retries in the face of failure, model-dependency analysis and concurrent scheduling of synchronization modules. Event-based synchronizers are simpler to implement, but lack the aforementioned guarantees. 

Difference between Work-based and Event-based Synchronizers

 

MechanismWork-based SynchronizersEvent-based Synchronizers
Control-logic BindingCheck if models are up-to-date based on their contentReact to events notifying of model updates
Implementation constraintsModules have to be idempotentModules are not required to be idempotent
DependenciesModules are executed in dependency order

Modules are executed reactively in an arbitrary order

ConcurrencyNon-dependent modules are executed concurrentlyModules are executed sequentially
Error HandlingErrors are propagated to dependencies, retries on failureNo error dependency. It's up to the Synchronizer to cope with event loss
Ease of implementationModerateEasy

Implementing an Event-based Synchronizer

Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service.

An Event-based Synchronizer is a collection of "Watcher" modules. Each Watcher module listens for ("watches") events pertaining to a particular model. The Synchronizer developer must provide the set of these modules. The steps for assembling a synchronizer once these modules have been implemented, are as follows:

  1. Run the gen_watcher.py script: gen_watcher.py <name of your app>
  2. Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to true. FIXME
  3. Install python-redis by running "pip install redis" in your Synchronizer container FIXME
  4. Link the redis container that comes packaged with xos with your Synchronizer container as "redis'" FIXME
  5. Drop your watcher modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps
  6. Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh

Watcher Module API

 TypeDescription
handle_watched_objectdef handle_watched_object(self, o)This method is called every time a watched object is added, deleted, or updated.
watch_degreeintDefines the set of watched models implicitly. If this module synchronizes models A and B, then the watched set is defined by the models that are a distance of watch_degree from A or from B in the model dependency graph.
watchedlist of type ModelLinkDefines the set of watched models explicitly. If this is defined, then watch_degree is ignored.
synchronizeslist of type ModelThe model that this module synchronizes.

The main body of a watcher module is the function handle_watched_object, which responds to operations on objects that the module synchronizes. If the module responds to multiple object types, then it must determine the type of object, and proceed to process it accordingly.

def handle_watched_object(self, o):
    if (type(o) is Slice):
        self.handle_changed_slice(o)
    elif (type(o) is Node):
        self.handle_changed_node(o)

Linking the Watcher into the Synchronizer

There are two ways of linking in a Watcher. Using them both does not hurt. The first method is complex but robust, and involves making the declaration in the data model, by ensuring that the model that your synchronizer would like to watch is linked to the model that it actuates. For instance, if your synchronizer actuates a service model called Fabric, which links the Instance model, then you would ensure that Instance is a dependency of Fabric by making the following annotation in the Fabric model:

class Fabric(Service):
    ...
    ...
    xos_links = [ModelLink(Instance,via='instance',into='ip')]

There can be several ModelLink specifications in a single xos_links declaration, each encapsulating the referenced model, the field in the current model that links to it, and the destination field in which the watcher is interested. If into is omitted, then the watcher is notified of all changes in the linked model, irrespective of the fields that change. 

The above change needs to be backed up with an instruction to the synchronizer that the watcher is interested in being notified of changes to its dependencies. This is done through a watch_degree annotation.

class SyncFabricService(SyncStep):
    watch_degree=1

By default, watch_degree is 0, meaning that the Synchronizer watches nothing. When watch degree is 1, it watches 1 level of dependencies removed, and so on. If the watch_degree in the above code were 2, then this module would also get notified of changes in dependencies of the Instance model.

The second way of linking in a watcher is to hardcode the watched model directly in the synchronizer:

class SyncFabricService(SyncStep):
    watched = [ModelLink(Instance,via='instance',into='ip')]

Linking the Watcher into the Synchronizer

  • Set the observer_enable_watchers option to true in your xos synchronizer config file
  • Add a link between your synchronizer container and the redis container by including the following lines in the definition of your synchronizer's docker-compose file. You may need to adapt these to the name of the project used (e.g. cordpod)
    • external_links: 
      • xos_redis:redis

  • Ensure that there is a similar link between your XOS UI container and the redis container

In addition to the above development tasks, you also need to make the following changes to your configuration to activate watchers.

Implementing a Work-based Synchronizer

Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service.

A work-based Synchronizer is a collection of "Actuator" modules. Each Actuator module is invoked when a model is found to be outdated relative to its last synchronization. An actuator module can be self-contained and written entirely in Python, or it can be broken into a "dispatcher" and "payload", with the dispatcher implemented in Python and the payload implemented using Ansible. The Synchronizer core has built-in support for the dispatch of Ansible modules and helps extract parameters from the synchronized model and translate them into the parameters required by the corresponding Ansible script. It also tracks an hierarchically structured list of such ansible scripts on the filesystem, for operators to use to inspect and debug a system. The procedure for building a work-based synchronizer is as follows:

  1. Run the gen_workbased.py script. gen_workbased <app name>.
  2. Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to False. FIXME
  3. Drop your actuator modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps
  4. Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh

Actuator Module API

 

 TypeDescription
synchronizeslist of type ModelThe set of models that the module synchronizes
sync_recorddef sync_record(self, object)A coarse-grained method that handles outdated objects.
delete_recorddef delete_record(self, object)A coarse-grained method that handles object deletion in the back end
get_extra_attributesdef get_extra_attributes(self, object)A method that maps an object to the parameters required by its Ansible payload. Returns a dict with those parameters and their values.
fetch_pendingdef fetch_pending(self, deleted)A method that fetches the set of pending objects from the database. The synchronizer core provides a default implementation. Override only if you have a reason to do so.
template_namestringThe name of the Ansible script that directly interacts with the underlying substrate.

Implementing a Step with Ansible

To implement a step using Ansible, a developer must provide two things: an Ansible recipe, and a get_extra_attributes method, which maps attributes of the object into a dictionary that configures that Ansible recipe. The Ansible recipe comes in two parts, an inner payload and a wrapper that delivers that payload to the VMs associated with the service. The wrapper itself comes in two parts. A part that sets up the preliminaries:

 

---
 
- hosts: "{{ instance_name }}"
  connection: ssh
  user: ubuntu
  sudo: yes
  gather_facts: no
  vars:
    - package_location: "{{ package_location }}"
    - server_port: "{{ server_port }}"

The template variables package_location and server_port come out of the Python part of the Synchronizer implementation (discussed shortly).

The outer wrapper then includes a set of Ansible roles that perform the required actions:

roles:
  - download_packages
  - configure_packages
  - start_server


The "payload" of the Ansible recipe contains an implementation of the roles, in this case, download_packages, configure_packages and start_server.

The concrete values of parameters required by the Ansible payload are provided in the implementation of the "get_extra_attributes" method in the Python part of the Synchronizer. This method receives an object from the data model and is charged with the task of converting the properties of that object into the set of properties required by the Ansible recipe, which are returned as a Python dictionary.

 

def get_extra_attributes(self, o):
        fields = {}
        fields['package_location'] = o.package_location
        fields['server_port'] = o.server_port
        return fields

 

Implementing a Step without Ansible

To implement a step without using Ansible, a developer need only implement the sync_record and delete_record methods, which get called for every pending object. These methods interact directly with the underlying substrate. 

Managing Dependencies

If your data models have dependencies between them, so that for one to be synchronized, another must already have been synchronized, then you can define such dependencies in your data model. The Synchronizer automatically picks up such dependencies and ensures that the steps corresponding to the models in questions are executed in a valid order. It also ensures that any errors that arise propagate from the affected objects to its dependents, and that the dependents are held up until the errors have been resolved and the dependencies have been successfully synchronized.

In the absence of failures, the Synchronizer tries to execute your synchronization steps concurrently to whatever extent this is possible while still honoring dependencies.

    <in the definition of your model>
    xos_links = [ModelLink(dest=MyServiceUser,via='user'),ModelLink(dest=MyServiceDevice,via='device')] 

In the above example, the xos_links field declares two dependencies. The name "xos_links" is key, and so the field should be named as such. The dependencies are contained in a list of type ModelLink, each of which defines a type of object (a model) and an "accessor" field via which a related object of that type can be accessed.

Handling Errors

To fault synchronization, you can raise an exception in any of the methods of your step that are automatically called by the synchronizer core. These include fetch_pending, sync_record and delete_record. The outcome of such exceptions has multiple parts:

  1. The synchronization of the present object is deferred.
  2. The synchronization of dependent objects is deferred, if those objects are accessible via the current object (see the 'via' field)
  3. A string representation of your exception is propagated into a scratchpad in your model, which in turn appears in your UI. When you click the object in question, in the UI, you should see the error message.
  4. The synchronization state of your object, and of dependent objects changes to "Error" and a red icon appears next to it.
  5. If the object repeatedly fails to synchronize, then its synchronization interval is increased exponentially.

Sometimes, you may encounter a temporary error, which you think may be resolved shortly, by the time the Synchronizer runs again. In these cases, you can raise a "DeferredException." This error type differs from a general exception in two ways:

  1. It does not put your object in error state
  2. It disabled exponential backoff - the Synchronizer tries to synchronize your object every single time.

Synchronizer Configuration Options

OptionDefaultPurpose

observer_disabled

FalseA directive to run the XOS server in Observer-less mode. Events are not relayed to the Observer and no bookkeeping is done.

observer_steps_dir

n/aThe path of the directory in which the Synchronizer will look for your watcher and actuator modules.

observer_sys_dir

n/aThe path of the directory that enlists backend objects your synchronizer creates. This is like the /sys directory in an operating system. Each entry is a file that contains an Ansible recipe that creates, updates or deletes your object. When you debug your synchronizer, you can run these files manually.

observer_pretend

FalseThis option runs the Synchronizer in "pretend" mode, in which synchronizer modules that use Ansible run in emulated mode, and do not actually execute backend API calls.

observer_proxy_ssh

n/a 

observer_name

n/aThe name of your Synchronizer. This is a required option.

observer_applist

coreA list consisting of the Django apps that your Synchronizer uses.

observer_dependency_graph

/opt/xos/model-depsDependencies between various models that your Synchronizer services. These are generated automatically by the Synchronizer utility "dmdot."

observer_backoff_disabled

TrueModels whose synchronization fails are re-executed, but with intervals that increase exponentially. This option disables the exponential growth of the intervals.

observer_logstash_hostport

n/aThe host name and port number (e.g. xosmonitor.org:4132) to which the Synchronizer streams its logs, on which a logstash server is running.

observer_log_file

n/aThe log file into which the Synchronizer logs are published.

observer_model_policies_dir

n/aThe directory in which model policies are stored.