Config/management model framework initial version
This is the initial commit of the internal config
model tree for Voltha.
A system that borrows key concepts from Docker, repo,
and git for multi-revision configuration handling with
transaction (commit/abort) logic, tagging, and the
ability to diff between config versions.
Key features:
* Stored model is defined using *annotated* Google protobuf
(*.proto) files
* Model is kept in memory as Python protobuf message objects
* The whole configuration is arranged in a nested (tree)
structure, made up of ConfigNode objects. Each
ConfigNode holds its config data as (possibly nested)
protobuf message object, as well as lists of "pointers"
to its logically nested children nodes. What message
fields are considered part of the node-local config vs.
what is stored as a child node is controlled by "child_node"
annotations in the *.proto files.
* Each ConifgNode stores its config in immutable
ConfigRevision obects, each revision being identified
by a unique hash value, calculated from a unique
hash value of its local configuration as well as
the list of hash values of all its children node.
* Collection type children nodes can be indexed (keyed)
so that they can be addressed with "path" notation
friendly to REST and other interfaces/APIs. Which
field is used as key is defined in the parent message
using "key" attribute of the "child_node" annotation.
Currently string and integer type fields can be used
as keys.
* Changes to the model create new revisions in all
affected nodes, which are rolled up as new revisions
to the root node.
* Root revisions can be tagged, tags can be moved
* The root node by default keeps a rev, but provides
a mechanism to purge untagged revs.
* All branch and leaf nodes auto-purge interim revs
not needed. A rev is not needed if no one refers
to it.
* Diffing between revs is supported, it yields RFC6902
jsonpatch objects. Diffing can be done between any
revs.
* The main operations are: CRUD (we call them .add,
.get, .update, .remove)
* Get can be recursive to an optionally limited depth.
* There is support for Read-Only attributes (fields)
* All CRUD operation support "path" based addressing.
* There is support for an branch/leaf node perspective
implemented by ConfigProxy. This proxy, when attached
to an arbitrary node in the tree, provides all the
CRUD operations in that context, that is, path args
are used relative to that node.
* Transaction support: All mutations made in a transaction
are invisible to others until the transaction is committed.
The commit is atomic (either all changes are applied
to the tree or none). Conflicts between transactions
are detected at the per-node level and a conflict
results in rejecting the conflicting transaction (first
one wins).
* Registered callbacks: via the proxy objects an
observer can register for pre- and post- operation
callbacks. Also, there is a post-get callback which
can be used to augment stored data with real-time
data.
I started hooking up the new config infrastructure to
Voltha's CORE, but this is still in progress, as not
all existing APIs have bee moved over yet.
Note: I also lumped in some experimental files working
with "Any" types in protobufs
Change-Id: Ic547b36e9b893d54e6d9ce67bdfcb32a6e8acd4c
diff --git a/common/utils/json_format.py b/common/utils/json_format.py
new file mode 100644
index 0000000..d1672e4
--- /dev/null
+++ b/common/utils/json_format.py
@@ -0,0 +1,92 @@
+"""
+Monkey patched json_format to allow best effort decoding of Any fields.
+Use the additional flag (strict_any_handling=False) to trigger the
+best-effort behavior. Omit the flag, or just use the original json_format
+module fot the strict behavior.
+"""
+
+from google.protobuf import json_format
+
+class _PatchedPrinter(json_format._Printer):
+
+ def __init__(self, including_default_value_fields=False,
+ preserving_proto_field_name=False,
+ strict_any_handling=False):
+ super(_PatchedPrinter, self).__init__(including_default_value_fields,
+ preserving_proto_field_name)
+ self.strict_any_handling = strict_any_handling
+
+ def _BestEffortAnyMessageToJsonObject(self, msg):
+ try:
+ res = self._AnyMessageToJsonObject(msg)
+ except TypeError:
+ res = self._RegularMessageToJsonObject(msg, {})
+ return res
+
+
+def MessageToDict(message,
+ including_default_value_fields=False,
+ preserving_proto_field_name=False,
+ strict_any_handling=False):
+ """Converts protobuf message to a JSON dictionary.
+
+ Args:
+ message: The protocol buffers message instance to serialize.
+ including_default_value_fields: If True, singular primitive fields,
+ repeated fields, and map fields will always be serialized. If
+ False, only serialize non-empty fields. Singular message fields
+ and oneof fields are not affected by this option.
+ preserving_proto_field_name: If True, use the original proto field
+ names as defined in the .proto file. If False, convert the field
+ names to lowerCamelCase.
+ strict_any_handling: If True, converion will error out (like in the
+ original method) if an Any field with value for which the Any type
+ is not loaded is encountered. If False, the conversion will leave
+ the field un-packed, but otherwise will continue.
+
+ Returns:
+ A dict representation of the JSON formatted protocol buffer message.
+ """
+ printer = _PatchedPrinter(including_default_value_fields,
+ preserving_proto_field_name,
+ strict_any_handling=strict_any_handling)
+ # pylint: disable=protected-access
+ return printer._MessageToJsonObject(message)
+
+
+def MessageToJson(message,
+ including_default_value_fields=False,
+ preserving_proto_field_name=False,
+ strict_any_handling=False):
+ """Converts protobuf message to JSON format.
+
+ Args:
+ message: The protocol buffers message instance to serialize.
+ including_default_value_fields: If True, singular primitive fields,
+ repeated fields, and map fields will always be serialized. If
+ False, only serialize non-empty fields. Singular message fields
+ and oneof fields are not affected by this option.
+ preserving_proto_field_name: If True, use the original proto field
+ names as defined in the .proto file. If False, convert the field
+ names to lowerCamelCase.
+ strict_any_handling: If True, converion will error out (like in the
+ original method) if an Any field with value for which the Any type
+ is not loaded is encountered. If False, the conversion will leave
+ the field un-packed, but otherwise will continue.
+
+ Returns:
+ A string containing the JSON formatted protocol buffer message.
+ """
+ printer = _PatchedPrinter(including_default_value_fields,
+ preserving_proto_field_name,
+ strict_any_handling=strict_any_handling)
+ return printer.ToJsonString(message)
+
+
+json_format._WKTJSONMETHODS['google.protobuf.Any'] = [
+ '_BestEffortAnyMessageToJsonObject',
+ '_ConvertAnyMessage'
+]
+
+json_format._Printer._BestEffortAnyMessageToJsonObject = \
+ json_format._Printer._AnyMessageToJsonObject