fixed XOS docs for move into Devel Guide
Change-Id: I1029bba94de5a1dd6d1f34202f01638fd8856a45
diff --git a/docs/README.md b/docs/README.md
index 19b9d53..69cf65a 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,4 +1,4 @@
-# Modeling Guide
+# Writing Models and Synchronizers
CORD adopts a model-based design, which is to say all aspects
of operating and managing CORD is mediated by a model-based
@@ -7,28 +7,35 @@
white paper:
[XOS: Modeling-as-a-Service](https://wiki.opencord.org/display/CORD/XOS+and+the+CORD+Controller?preview=/1279317/4981376/XOS%20Modeling-as-a-Service.pdf).
-This guide describes XOS, and the role it plays in implementing CORD's
-control plane. XOS is not a monolithic component. It is best viewed as
-having three inter-related aspects, and this guide is organized accordingly.
+XOS has three inter-related aspects, and this section is
+organized accordingly.
-First, XOS defines a [modeling framework](dev/xproto.md), which
-includes both a modeling language (*xproto*) and a generative
-toolchain (*xosgenx*). The core abstractions that define CORD's
-behavior are expressed in xproto, with xosgenx then used to
-generate code for several elements required to control CORD
-(including an API that serves the set of models that have been
-loaded into XOS).
+* **Modeling Framework:** XOS defines a
+ [modeling framework](dev/xproto.md), which
+ includes both a modeling language (*xproto*) and a generative
+ toolchain (*xosgenx*). The abstractions that define CORD's
+ behavior are expressed in xproto, with xosgenx then used to
+ generate code for several elements required to control CORD
+ (including an API that serves the set of models that have been
+ loaded into XOS). Service developers typically write one or more
+ models to on-board their service.
-Second, CORD is based on a core set of models. These models are
-expressed and realized using XOS, but they are architecturally
-independent of XOS. These models are central to defining what
-CORD **is**, including its [core abstractions](core_models.md)
-and the [security policies](security_policies.md) that govern how
-various principals can act on those abstractions in a multi-tenant
-environment.
+* **Synchronizer Framework:** XOS defines a
+ [synchronization framework](dev/synchronizers.md)
+ that actuates the CORD data model. This framework is reponsible for
+ driving the underlying components configured into CORD (for example,
+ services, access devices) towards the desired state. Service developers
+ typically write a synchronizer to on-board their services.
+
+* **Core Models and Policies:** The CORD platform is defined by
+ a [core](core_models.md) set of *xproto* models, plus a set of
+ [security policies](security_policies.md) that govern how
+ various principals can act on those models in a multi-tenant
+ environment. Platform developers typically define and evolve the
+ core models and policies, which effectively establishes the
+ foundation on which all services run and are interconnected into
+ service graphs.
-Third, XOS defines a [synchronization framework](dev/synchronizers.md)
-that actuates the CORD data model. This framework is reponsible for
-driving the underlying components configured into CORD (for example,
-services, access devices) towards the desired state.
+
+
diff --git a/docs/dev/sync_arch.md b/docs/dev/sync_arch.md
index 6fe14bb..5ab8770 100644
--- a/docs/dev/sync_arch.md
+++ b/docs/dev/sync_arch.md
@@ -1,4 +1,4 @@
-# Synchronizer Design Guidelines
+# Synchronizer Design
Synchronizers act as the link between the data model and the functional half of
the system. The data model contains a clean, abstract and declarative
diff --git a/docs/dev/sync_impl.md b/docs/dev/sync_impl.md
index b0ca5d3..613bb57 100644
--- a/docs/dev/sync_impl.md
+++ b/docs/dev/sync_impl.md
@@ -1,4 +1,4 @@
-# Synchronizer Implementation Details
+# Synchronizer Implementation
There are three types of synchronizers: _Work-based_, _Event-based_, and
_Hybrid_ (the last of which subsume the functionalities of the first two).
@@ -119,7 +119,7 @@
watched = [ModelLink(Instance,via='instance',into='ip')]
```
-#### Activate the Watcher by connecting the Synchronzier Container to Redis
+#### Activate the Watcher by Connecting to Redis
* Set the `observer_enable_watchers` option to true in your XOS synchronizer
config file.
diff --git a/docs/dev/synchronizers.md b/docs/dev/synchronizers.md
index 9dff5de..cfb8ce7 100644
--- a/docs/dev/synchronizers.md
+++ b/docs/dev/synchronizers.md
@@ -1,4 +1,4 @@
-# Writing Synchronizers
+# XOS Synchronizer Framework
Synchronizers are the components of CORD that map the abstract declarative
state about how the system is suppose to behave (as defined by the XOS data
diff --git a/docs/dev/xosgenx.md b/docs/dev/xosgenx.md
new file mode 100644
index 0000000..a6abc1e
--- /dev/null
+++ b/docs/dev/xosgenx.md
@@ -0,0 +1,158 @@
+# XOS Tool Chain (Internals)
+
+The xosgenx tool converts a xproto file into an intermediate representation and
+passes it to a target, which in turn generates the output code. The target has
+access to a library of auxiliary functions implemented in Python. The target
+itself is written as a jinja2 template. The following figure depicts the
+processing pipeline. Developers will not need to interact with xosgenx
+directly, except when there is an opportunity to create a new target.
+
+
+
+## Intermediate Representation (IR)
+
+The IR is a representation of a parsed xproto file in the form of nested Python
+dictionaries. Here is a description of its structure.
+
+```protobuf
+"proto": {
+ "messages": [
+ {"name": "foo", fields: [{...}], links: [{...}], rlinks: [{...}], options: [{...}]}
+ ]
+},
+"context": {
+ "command line option 1": "value - see the --kv option of xosgenx"
+},
+"options": {
+ "top level option 1": "value of option 1"
+}
+```
+
+## Library Functions
+
+xproto targets can use a set of library functions implemented in Python. These
+can be found in the file `lib.py` in the `genx/tool` directory. These functions
+are listed below:
+
+* `xproto_unquote(string)` Unquotes a string. For example, `"This is a help
+ string"` is converted into `This is a help string.`
+
+* `xproto_singularize(field)` Converts an English plural into its singular. It
+ is extracted from the `singular` option for a field if such an option is
+ specified. Otherwise, it performs the conversion automatically using the
+ library `pattern.en`.
+
+* `xproto_pluralize(field)` The reverse of `xproto_singularize`.
+
+## Targets
+
+A target is a template written in jinja2 that takes the IR as input and
+generates code (a text file) as output. Common targets are Python, Protobufs,
+unit tests, and so on. The following example shows how to generate a
+GraphViz dot file from a set of xproto specifications:
+
+```python
+digraph {
+{% for m in proto.messages -%}
+ {%- for l in m.links %}
+ {{ m.name }} -> {{ l.peer }};
+ {%- endfor %}
+{% endfor %}
+}
+```
+
+This template loops through all of the messages in a proto definition and then
+through the links in each message. For each link, it formats and outputs an
+edge in a graph in Graphviz dot notation.
+
+```python
+{{ proto }}
+```
+
+This target simply prints the IR for an xproto definition.
+
+```python
+{% for m in proto.messages -%}
+{% for r in m.rlinks %}
+ def enumerate_{{ xos_singularize(r) }}_ids:
+ return map(lambda x:x['id'], {{ xos_pluralize(r) }})
+{% endfor %}
+{% endfor -%}
+```
+
+The example target outputs a Python function that enumerates the ids of the
+objects from which the current object is linked.
+
+## Running xosgenx
+
+It is possible to run the xosgenx tool chain directly. This is useful, for
+example, when developing a new target.
+
+To do do, first setup the python virtual environment as described
+[here](local_env.md). Then drop an xproto file in your working directory. For
+example, you can copy-and-paste the following content into a file named
+`slice.xproto`:
+
+```protobuf
+message Slice::slice_policy (XOSBase) {
+ option validators = "slice_name:Slice name ({obj.name}) must begin with site login_base ({ obj.site.login_base}), slice_name_length_and_no_spaces:Slice name too short or contains spaces, slice_has_creator:Slice has no creator";
+ option plural = "Slices";
+
+ required string name = 1 [max_length = 80, content_type = "stripped", blank = False, help_text = "The Name of the Slice", null = False, db_index = False];
+ required bool enabled = 2 [help_text = "Status for this Slice", default = True, null = False, db_index = False, blank = True];
+ required string description = 4 [help_text = "High level description of the slice and expected activities", max_length = 1024, null = False, db_index = False, blank = True, varchar = True];
+ required string slice_url = 5 [db_index = False, max_length = 512, null = False, content_type = "url", blank = True];
+ required manytoone site->Site:slices = 6 [help_text = "The Site this Slice belongs to", null = False, db_index = True, blank = False];
+ required int32 max_instances = 7 [default = 10, null = False, db_index = False, blank = False];
+ optional manytoone service->Service:slices = 8 [db_index = True, null = True, blank = True];
+ optional string network = 9 [blank = True, max_length = 256, null = True, db_index = False, choices = "((None, 'Default'), ('host', 'Host'), ('bridged', 'Bridged'), ('noauto', 'No Automatic Networks'))"];
+ optional string exposed_ports = 10 [db_index = False, max_length = 256, null = True, blank = True];
+ optional manytoone creator->User:slices = 12 [db_index = True, null = True, blank = True];
+ optional manytoone default_flavor->Flavor:slices = 13 [db_index = True, null = True, blank = True];
+ optional manytoone default_image->Image:slices = 14 [db_index = True, null = True, blank = True];
+ optional manytoone default_node->Node:slices = 15 [db_index = True, null = True, blank = True];
+ optional string mount_data_sets = 16 [default = "GenBank", max_length = 256, content_type = "stripped", blank = True, null = True, db_index = False];
+ required string default_isolation = 17 [default = "vm", choices = "(('vm', 'Virtual Machine'), ('container', 'Container'), ('container_vm', 'Container In VM'))", max_length = 30, blank = False, null = False, db_index = False];
+}
+```
+
+One of the existing targets is Django, which currently serves as the
+Object-Relational Mapping (ORM) tool used in CORD. To generate a Django model
+starting from this xproto file you can use:
+
+```shell
+xosgenx --target="django.xtarget" --output=. --write-to-file="model" --dest-extension="py" slice.xproto
+```
+
+This generates a file called `slice.py` in your current directory. If there
+were multiple files, then it generates python Django models for each of them.
+
+You can print the tool’s syntax by running `xosgenx --help`.
+
+```shell
+usage: xosgenx [-h] [--rev] --target TARGET [--output OUTPUT] [--attic ATTIC]
+ [--kvpairs KV] [--write-to-file {single,model,target}]
+ [--dest-file DEST_FILE | --dest-extension DEST_EXTENSION]
+ <input file> [<input file> ...]
+
+XOS Generative Toolchain
+
+positional arguments:
+ <input file> xproto files to compile
+
+optional arguments:
+ -h, --help show this help message and exit
+ --rev Convert proto to xproto
+ --target TARGET Output format, corresponding to <output>.yaml file
+ --output OUTPUT Destination dir
+ --attic ATTIC The location at which static files are stored
+ --kvpairs KV Key value pairs to make available to the target
+ --write-to-file {single,model,target}
+ Single output file (single) or output file per model
+ (model) or let target decide (target)
+ --dest-file DEST_FILE
+ Output file name (if write-to-file is set to single)
+ --dest-extension DEST_EXTENSION
+ Output file extension (if write-to-file is set to
+ single)
+```
diff --git a/docs/dev/xproto.md b/docs/dev/xproto.md
index 30e5e1d..64b327b 100644
--- a/docs/dev/xproto.md
+++ b/docs/dev/xproto.md
@@ -443,161 +443,4 @@
* sub-policy reference ( `* <policy name>` ),
* python escapes ({% raw %}`{{ python expression }}`{% endraw %}).
-## Tool Chain
-
-The xosgenx tool converts a xproto file into an intermediate representation and
-passes it to a target, which in turn generates the output code. The target has
-access to a library of auxiliary functions implemented in Python. The target
-itself is written as a jinja2 template. The following figure depicts the
-processing pipeline.
-
-
-
-### Intermediate Representation (IR)
-
-The IR is a representation of a parsed xproto file in the form of nested Python
-dictionaries. Here is a description of its structure.
-
-```protobuf
-"proto": {
- "messages": [
- {"name": "foo", fields: [{...}], links: [{...}], rlinks: [{...}], options: [{...}]}
- ]
-},
-"context": {
- "command line option 1": "value - see the --kv option of xosgenx"
-},
-"options": {
- "top level option 1": "value of option 1"
-}
-```
-
-### Library Functions
-
-xproto targets can use a set of library functions implemented in Python. These
-can be found in the file `lib.py` in the `genx/tool` directory. These functions
-are listed below:
-
-* `xproto_unquote(string)` Unquotes a string. For example, `"This is a help
- string"` is converted into `This is a help string.`
-
-* `xproto_singularize(field)` Converts an English plural into its singular. It
- is extracted from the `singular` option for a field if such an option is
- specified. Otherwise, it performs the conversion automatically using the
- library `pattern.en`.
-
-* `xproto_pluralize(field)` The reverse of `xproto_singularize`.
-
-### Targets
-
-A target is a template written in jinja2 that takes the IR as input and
-generates code (a text file) as output. Common targets are Python, Protobufs,
-unit tests, and so on. The following example shows how to generate a GraphViz
-dot file from a set of xproto specifications:
-
-```python
-digraph {
-{% for m in proto.messages -%}
- {%- for l in m.links %}
- {{ m.name }} -> {{ l.peer }};
- {%- endfor %}
-{% endfor %}
-}
-```
-
-This template loops through all of the messages in a proto definition and then
-through the links in each message. For each link, it formats and outputs an
-edge in a graph in Graphviz dot notation.
-
-```python
-{{ proto }}
-```
-
-This target simply prints the IR for an xproto definition.
-
-```python
-{% for m in proto.messages -%}
-{% for r in m.rlinks %}
- def enumerate_{{ xos_singularize(r) }}_ids:
- return map(lambda x:x['id'], {{ xos_pluralize(r) }})
-{% endfor %}
-{% endfor -%}
-```
-
-The example target outputs a Python function that enumerates the ids of the
-objects from which the current object is linked.
-
-### Running xosgenx
-
-It is possible to run the xosgenx tool chain directly. This is useful, for
-example, when developing a new target.
-
-To do do, first setup the python virtual environment as described
-[here](local_env.md). Then drop an xproto file in your working directory. For
-example, you can copy-and-paste the following content into a file named
-`slice.xproto`:
-
-```protobuf
-message Slice::slice_policy (XOSBase) {
- option validators = "slice_name:Slice name ({obj.name}) must begin with site login_base ({ obj.site.login_base}), slice_name_length_and_no_spaces:Slice name too short or contains spaces, slice_has_creator:Slice has no creator";
- option plural = "Slices";
-
- required string name = 1 [max_length = 80, content_type = "stripped", blank = False, help_text = "The Name of the Slice", null = False, db_index = False];
- required bool enabled = 2 [help_text = "Status for this Slice", default = True, null = False, db_index = False, blank = True];
- required string description = 4 [help_text = "High level description of the slice and expected activities", max_length = 1024, null = False, db_index = False, blank = True, varchar = True];
- required string slice_url = 5 [db_index = False, max_length = 512, null = False, content_type = "url", blank = True];
- required manytoone site->Site:slices = 6 [help_text = "The Site this Slice belongs to", null = False, db_index = True, blank = False];
- required int32 max_instances = 7 [default = 10, null = False, db_index = False, blank = False];
- optional manytoone service->Service:slices = 8 [db_index = True, null = True, blank = True];
- optional string network = 9 [blank = True, max_length = 256, null = True, db_index = False, choices = "((None, 'Default'), ('host', 'Host'), ('bridged', 'Bridged'), ('noauto', 'No Automatic Networks'))"];
- optional string exposed_ports = 10 [db_index = False, max_length = 256, null = True, blank = True];
- optional manytoone creator->User:slices = 12 [db_index = True, null = True, blank = True];
- optional manytoone default_flavor->Flavor:slices = 13 [db_index = True, null = True, blank = True];
- optional manytoone default_image->Image:slices = 14 [db_index = True, null = True, blank = True];
- optional manytoone default_node->Node:slices = 15 [db_index = True, null = True, blank = True];
- optional string mount_data_sets = 16 [default = "GenBank", max_length = 256, content_type = "stripped", blank = True, null = True, db_index = False];
- required string default_isolation = 17 [default = "vm", choices = "(('vm', 'Virtual Machine'), ('container', 'Container'), ('container_vm', 'Container In VM'))", max_length = 30, blank = False, null = False, db_index = False];
-}
-```
-
-One of the existing targets is Django, which currently serves as the
-Object-Relational Mapping (ORM) tool used in CORD. To generate a Django model
-starting from this xproto file you can use:
-
-```shell
-xosgenx --target="django.xtarget" --output=. --write-to-file="model" --dest-extension="py" slice.xproto
-```
-
-This generates a file called `slice.py` in your current directory. If there
-were multiple files, then it generates python Django models for each of them.
-
-You can print the tool’s syntax by running `xosgenx --help`.
-
-```shell
-usage: xosgenx [-h] [--rev] --target TARGET [--output OUTPUT] [--attic ATTIC]
- [--kvpairs KV] [--write-to-file {single,model,target}]
- [--dest-file DEST_FILE | --dest-extension DEST_EXTENSION]
- <input file> [<input file> ...]
-
-XOS Generative Toolchain
-
-positional arguments:
- <input file> xproto files to compile
-
-optional arguments:
- -h, --help show this help message and exit
- --rev Convert proto to xproto
- --target TARGET Output format, corresponding to <output>.yaml file
- --output OUTPUT Destination dir
- --attic ATTIC The location at which static files are stored
- --kvpairs KV Key value pairs to make available to the target
- --write-to-file {single,model,target}
- Single output file (single) or output file per model
- (model) or let target decide (target)
- --dest-file DEST_FILE
- Output file name (if write-to-file is set to single)
- --dest-extension DEST_EXTENSION
- Output file extension (if write-to-file is set to
- single)
-```