first edit pass through new docs
Change-Id: I8e1651fd86e6b9cc99810272e1b0204558263f1b
diff --git a/docs/README.md b/docs/README.md
index afc3c54..b496cc6 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -3,11 +3,13 @@
CORD adopts a model-based design, which is to say all aspects
of operating and managing CORD is mediated by a model-based
control plane. XOS is the component in CORD that implements
-this control plane.
+this control plane. For an overview of XOS, see the following
+white paper:
+[XOS: Modeling-as-a-Service](https://wiki.opencord.org/display/CORD/XOS%3A+The+CORD+Controller?preview=/1279317/4981376/XOS%20Modeling-as-a-Service.pdf).
-This guide describes XOS, and the role it plays in the CORD Controller.
-XOS is not a monolithic component. It is best viewed as having
-three inter-related aspects, and this guide is organized accordingly.
+This guide describes XOS, and the role it plays in implementing CORD's
+control plane. XOS is not a monolithic component. It is best viewed as
+having three inter-related aspects, and this guide is organized accordingly.
First, XOS defines a [modeling framework](dev/xproto.md), which
includes both a modeling language (*xproto*) and a generative
diff --git a/docs/dev/sync_reference.md b/docs/dev/sync_reference.md
new file mode 100644
index 0000000..4bab382
--- /dev/null
+++ b/docs/dev/sync_reference.md
@@ -0,0 +1,265 @@
+# Synchronizer Reference
+
+This section is a reference for the commonly used APIs exposed
+by the synchronizer framework.
+
+## Convenience Methods
+
+As part of the model definition, it is possible to extend the autogenerated gRPC
+APIs with custom methods, for example, to facilitate access to some kind of data
+from the synchronizers.
+
+`convenience_methods` can be defined in a folder (`convenience`) that needs to
+be a child of the `models_dir` as per your configuration.
+
+> For example if your configuration contains:
+>
+> ```yaml
+> models_dir: "/opt/xos/synchronizers/<synchronizer_name>/models"
+> ```
+>
+> then you `convenience methods` needs to be located in
+> `/opt/xos/synchronizers/<synchronizer_name>/models/convenience`
+
+Assuming our model definition looks like:
+
+```proto
+message MyModel (XOSBase){
+ required string first_name = 1 [null = False, blank = False];
+ required string last_name = 2 [null = False, blank = False];
+}
+```
+
+here is an example of a basic convenience methods that will expose a
+`full_name` property over the APIs used by the synchronizers:
+
+```python
+from xosapi.orm import ORMWrapper, register_convenience_wrapper
+
+from xosconfig import Config
+from multistructlog import create_logger
+
+log = create_logger(Config().get('logging'))
+
+class ORMWrapperMyModel(ORMWrapper):
+
+ @property
+ def full_name(self):
+ return "%s %s" % (self.first_name, self.last_name)
+
+register_convenience_wrapper("MyModel", ORMWrapperMyModel)
+```
+
+> **Note:** The convenience methods will be loaded in all the synchronizer
+> containers so that they can be used in multiple places.
+
+## Model Policies
+
+Model Policies can be seen as `post-save` hooks and they are generally defined
+in the `xos/synchronizer/model_policies` folder of your service.
+
+Model policies are generally used to dynamically create a service chain (when a
+ServiceInstance is created it will create a ServiceInstance of its east side
+Service)
+
+> **Note:** You'll need to add this folder in your synchronizer configuration file
+> as:
+>
+> ```yaml
+> model_policies_dir: "/opt/xos/synchronizers/<synchronizer_name>/model_policies"
+> ```
+
+A model policy is a class that inherits from `Policy`:
+
+```python
+from synchronizers.new_base.modelaccessor import MyServiceInstance, ServiceInstanceLink, model_accessor
+from synchronizers.new_base.policy import Policy
+
+class MyServiceInstancePolicy(Policy):
+ model_name = "MyServiceInstance"
+```
+
+and overrides one or more of the following methods:
+
+```python
+def handle_create(self, model):
+```
+
+```python
+def handle_update(self, model):
+```
+
+```python
+def handle_delete(self, model):
+```
+
+Where `model` is the instance of the model that has been created.
+
+## Sync Steps
+
+Sync Steps are the actual piece of code that provide the mapping between your
+models and your backend. You will need to define a sync step for each model.
+
+> **Note:** You'll need to add this folder in your synchronizer configuration file
+> as:
+>
+> ```yaml
+> steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/steps"
+> ```
+
+A Sync Step is a class that inherits from `SyncStep`:
+
+```python
+
+from synchronizers.new_base.SyncInstanceUsingAnsible import SyncStep
+from synchronizers.new_base.modelaccessor import MyModel
+
+from xosconfig import Config
+from multistructlog import create_logger
+
+log = create_logger(Config().get('logging'))
+
+class SyncMyModel(SyncStep):
+ provides = [MyModel]
+
+ observes = MyModel
+```
+
+and provides these methods:
+
+```python
+def sync_record(self, o):
+ log.info("sync'ing object", object=str(o), **o.tologdict())
+```
+
+```python
+def delete_record(self, o):
+ log.info("deleting object", object=str(o), **o.tologdict())
+```
+
+This methods will be invoked anytime there is change in the model passing as
+argument the changed models. After performing the required operations to sync
+the model state with the backend state the synchronizer framework will update
+the models with the operational informations needed.
+
+## Pull Steps
+
+Pull Steps can be used to observe the surrounding environment and update the
+data-model accordingly.
+
+> **Note:** You'll need to add this folder in your synchronizer configuration file
+> as:
+>
+> ```yaml
+> pull_steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/pull_steps"
+> ```
+
+A Sync Step is a class that inherits from `PullStep`
+
+```python
+from synchronizers.new_base.pullstep import PullStep
+from synchronizers.new_base.modelaccessor import OLTDevice
+
+from xosconfig import Config
+from multistructlog import create_logger
+
+log = create_logger(Config().get('logging'))
+
+from synchronizers.new_base.modelaccessor import MyModel
+
+class MyModelPullStep(PullStep):
+ def __init__(self):
+ super(MyModelPullStep, self).__init__(observed_model=OLTDevice)
+```
+
+and override the following method:
+
+```python
+def pull_records(self):
+ log.info("pulling MyModels")
+ # create an empty model
+ o = MyModel()
+ # code to fetch information
+ # populate the model
+ o.first_name = "John"
+ o.last_name = "Doe"
+ o.no_sync = True # this is required to prevent the synchronizer to be invoked and start a loop
+ o.save()
+```
+
+## Event Steps
+
+Event Steps are similar to pull steps in that they are often used to implement a flow of information from the environment into the data model. However, rather than using polling, event steps rely on externally generated events delivered via an event bus, such as Kafka.
+
+> **Note:** You'll need to add this folder in your synchronizer configuration file
+> as:
+>
+> ```yaml
+> event_steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/event_steps"
+> ```
+>
+> You'll also need to make sure the event bus endpoint is specified in the
+> synchronizer config file. For example:
+>
+> ```yaml
+> event_bus:
+> endpoint: cord-kafka-kafka
+> kind: kafka
+>```
+
+An event step inherits from the `EventStep` class:
+
+```python
+
+import json
+from synchronizers.new_base.eventstep import EventStep
+from synchronizers.new_base.modelaccessor import MyModel
+from xosconfig import Config
+from multistructlog import create_logger
+
+log = create_logger(Config().get('logging'))
+
+class MyModelEventStep(EventStep):
+ technology = "kafka"
+ topics = ["MyTopic"]
+
+ def __init__(self, *args, **kwargs):
+ super(MyEventStep, self).__init__(*args, **kwargs)
+```
+
+Two important class members that are defined in each event step are `technology` and `topics`. `technology` tells what type of event bus to use. There's currently only one bus interface implemented by the synchronizer framework, and that is `kafka`. The `topics` member is a list of topics that will be listened on for events. The precise meaning of `topics` is left to the particular event bus technology that is in use.
+
+Service-specific logic is implemented by overriding the `process_event()` method:
+
+```python
+
+ def process_event(self, event):
+ value = json.loads(event.value)
+ first_name = value["first_name"]
+ last_name = value["last_name"]
+
+ # See if the object already exists
+ objs = MyModel.filter(first_name=first_name, last_name=last_name)
+ if objs:
+ return
+
+ # Create a new object
+ obj = MyModel()
+ obj.first_name = first_name
+ obj.last_name = last_name
+ obj.save(always_update_timestamp = True)
+```
+
+In this example we've made the assumption that the value of an event is a json-encoded dictionary containing the keys `first_name` and `last_name`. The event step in this case checks to see if an object with those fields already exists, and if not then it creates the object.
+
+In this example, we've differed from the Pull Step example in that we omitted `no_sync=True` and we added `always_update_timestamp = True` to the `save()` call. This has the effect of causing the synchronizer to excute any sync steps that might exist for `MyModel`. Whether or not you want sync_steps to run is an implementation decision and depends upon the design of your synchronizer.
+
+Sending an event to Kafka can be done using a variety of Kafka clients for various languages, Kafka command-line tools, etc. A python example is as follows:
+
+```python
+import json
+from kafka import KafkaProducer
+producer = KafkaProducer(bootstrap_servers="cord-kafka-kafka")
+producer.send("MyTopic", json.dumps({"first_name": "John", "last_name": "Doe"}))
+producer.flush()
+```
diff --git a/docs/dev/synchronizers.md b/docs/dev/synchronizers.md
index e8bf768..9dff5de 100644
--- a/docs/dev/synchronizers.md
+++ b/docs/dev/synchronizers.md
@@ -10,273 +10,13 @@
specification, and then you implement a synchronizer that translates that model
onto some backend component.
-To implement a Synchronizer, it is important to first understand the role they
-play in CORD and the assumptions made about their behavior. The next section
-describes a set of [design guidelines](sync_arch.md) and the following one
-presents the [implementation details](sync_impl.md).
-
-## Overview of the synchronizer framework APIs
-
-This section is intended to be a reference for the commonly used APIs exposed
-by the synchronizer framework.
-
-### Convenience Methods
-
-As part of the model definition is possible to extend the autogenerated gRPC
-APIs with custom methods, for example to facilitate access to some kind of data
-from the synchronizers.
-
-`convenience_methods` can be defined in a folder (`convenience`) that needs to
-be a child of the `models_dir` as per your configuration.
-
-> For example if your configuration contains:
->
-> ```yaml
-> models_dir: "/opt/xos/synchronizers/<synchronizer_name>/models"
-> ```
->
-> then you `convenience methods` needs to be located in
-> `/opt/xos/synchronizers/<synchronizer_name>/models/convenience`
-
-Assuming our model definition looks like:
-
-```proto
-message MyModel (XOSBase){
- required string first_name = 1 [null = False, blank = False];
- required string last_name = 2 [null = False, blank = False];
-}
-```
-
-here is an example of a basic convenience methods that will expose a
-`full_name` property over the APIs used by the synchronizers:
-
-```python
-from xosapi.orm import ORMWrapper, register_convenience_wrapper
-
-from xosconfig import Config
-from multistructlog import create_logger
-
-log = create_logger(Config().get('logging'))
-
-class ORMWrapperMyModel(ORMWrapper):
-
- @property
- def full_name(self):
- return "%s %s" % (self.first_name, self.last_name)
-
-register_convenience_wrapper("MyModel", ORMWrapperMyModel)
-```
-
-> NOTE: The convenience methods will be loaded in all the synchronizer
-> containers so that they can be used in multiple places.
-
-### Model Policies
-
-Model Policies can be seen as `post-save` hooks and they are generally defined
-in the `xos/synchronizer/model_policies` folder of your service.
-
-Model policies are generally used to dynamically create a service chain (when a
-ServiceInstance is created it will create a ServiceInstance of its east side
-Service)
-
-> NOTE: You'll need to add this folder in your synchronizer configuration file
-> as:
->
-> ```yaml
-> model_policies_dir: "/opt/xos/synchronizers/<synchronizer_name>/model_policies"
-> ```
-
-A model policy is a class that inherits from `Policy`:
-
-```python
-from synchronizers.new_base.modelaccessor import MyServiceInstance, ServiceInstanceLink, model_accessor
-from synchronizers.new_base.policy import Policy
-
-class MyServiceInstancePolicy(Policy):
- model_name = "MyServiceInstance"
-```
-
-and overrides one or more of the following methods:
-
-```python
-def handle_create(self, model):
-```
-
-```python
-def handle_update(self, model):
-```
-
-```python
-def handle_delete(self, model):
-```
-
-Where `model` is the instance of the model that has been created.
-
-### Sync Steps
-
-Sync Steps are the actual piece of code that provide the mapping between your
-models and your backend. You will need to define a sync step for each model.
-
-> NOTE: You'll need to add this folder in your synchronizer configuration file
-> as:
->
-> ```yaml
-> steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/steps"
-> ```
-
-A Sync Step is a class that inherits from `SyncStep`:
-
-```python
-
-from synchronizers.new_base.SyncInstanceUsingAnsible import SyncStep
-from synchronizers.new_base.modelaccessor import MyModel
-
-from xosconfig import Config
-from multistructlog import create_logger
-
-log = create_logger(Config().get('logging'))
-
-class SyncMyModel(SyncStep):
- provides = [MyModel]
-
- observes = MyModel
-```
-
-and provides these methods:
-
-```python
-def sync_record(self, o):
- log.info("sync'ing object", object=str(o), **o.tologdict())
-```
-
-```python
-def delete_record(self, o):
- log.info("deleting object", object=str(o), **o.tologdict())
-```
-
-This methods will be invoked anytime there is change in the model passing as
-argument the changed models. After performing the required operations to sync
-the model state with the backend state the synchronizer framework will update
-the models with the operational informations needed.
-
-### Pull Steps
-
-Pull Steps can be used to observe the surrounding environment and update the
-data-model accordingly.
-
-> NOTE: You'll need to add this folder in your synchronizer configuration file
-> as:
->
-> ```yaml
-> pull_steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/pull_steps"
-> ```
-
-A Sync Step is a class that inherits from `PullStep`
-
-```python
-from synchronizers.new_base.pullstep import PullStep
-from synchronizers.new_base.modelaccessor import OLTDevice
-
-from xosconfig import Config
-from multistructlog import create_logger
-
-log = create_logger(Config().get('logging'))
-
-from synchronizers.new_base.modelaccessor import MyModel
-
-class MyModelPullStep(PullStep):
- def __init__(self):
- super(MyModelPullStep, self).__init__(observed_model=OLTDevice)
-```
-
-and override the following method:
-
-```python
-def pull_records(self):
- log.info("pulling MyModels")
- # create an empty model
- o = MyModel()
- # code to fetch information
- # populate the model
- o.first_name = "John"
- o.last_name = "Doe"
- o.no_sync = True # this is required to prevent the synchronizer to be invoked and start a loop
- o.save()
-```
-
-### Event Steps
-
-Event Steps are similar to pull steps in that they are often used to implement a flow of information from the environment into the data model. However, rather than using polling, event steps rely on externally generated events delivered via an event bus, such as Kafka.
-
-> NOTE: You'll need to add this folder in your synchronizer configuration file
-> as:
->
-> ```yaml
-> event_steps_dir: "/opt/xos/synchronizers/<synchronizer_name>/event_steps"
-> ```
->
-> You'll also need to make sure the event bus endpoint is specified in the
-> synchronizer config file. For example:
->
-> ```yaml
-> event_bus:
-> endpoint: cord-kafka-kafka
-> kind: kafka
->```
-
-An event step inherits from the `EventStep` class:
-
-```python
-
-import json
-from synchronizers.new_base.eventstep import EventStep
-from synchronizers.new_base.modelaccessor import MyModel
-from xosconfig import Config
-from multistructlog import create_logger
-
-log = create_logger(Config().get('logging'))
-
-class MyModelEventStep(EventStep):
- technology = "kafka"
- topics = ["MyTopic"]
-
- def __init__(self, *args, **kwargs):
- super(MyEventStep, self).__init__(*args, **kwargs)
-```
-
-Two important class members that are defined in each event step are `technology` and `topics`. `technology` tells what type of event bus to use. There's currently only one bus interface implemented by the synchronizer framework, and that is `kafka`. The `topics` member is a list of topics that will be listened on for events. The precise meaning of `topics` is left to the particular event bus technology that is in use.
-
-Service-specific logic is implemented by overriding the `process_event()` method:
-
-```python
-
- def process_event(self, event):
- value = json.loads(event.value)
- first_name = value["first_name"]
- last_name = value["last_name"]
-
- # See if the object already exists
- objs = MyModel.filter(first_name=first_name, last_name=last_name)
- if objs:
- return
-
- # Create a new object
- obj = MyModel()
- obj.first_name = first_name
- obj.last_name = last_name
- obj.save(always_update_timestamp = True)
-```
-
-In this example we've made the assumption that the value of an event is a json-encoded dictionary containing the keys `first_name` and `last_name`. The event step in this case checks to see if an object with those fields already exists, and if not then it creates the object.
-
-In this example, we've differed from the Pull Step example in that we omitted `no_sync=True` and we added `always_update_timestamp = True` to the `save()` call. This has the effect of causing the synchronizer to excute any sync steps that might exist for `MyModel`. Whether or not you want sync_steps to run is an implementation decision and depends upon the design of your synchronizer.
-
-Sending an event to Kafka can be done using a variety of Kafka clients for various languages, Kafka command-line tools, etc. A python example is as follows:
+To implement a Synchronizer, it is important to first understand the role
+they play in CORD and the assumptions made about their behavior. The
+following three subsections address these issues. The first presents
+a set of [design guidelines](sync_arch.md) for how to write synchronizers,
+the second describes some of the [implementation details](sync_impl.md)
+about synchronizer internals, and the third gives a reference definition of
+the [framework API](sync_reference.md) for programming synchronizers.
+Developers that already understand the basics of synchronizers might
+want to jump to the third subsection for programming details.
-```python
-import json
-from kafka import KafkaProducer
-producer = KafkaProducer(bootstrap_servers="cord-kafka-kafka")
-producer.send("MyTopic", json.dumps({"first_name": "John", "last_name": "Doe"}))
-producer.flush()
-```
\ No newline at end of file