VOL-724 VOL-2106 Return of reconciliation and MIB external storage

Requires pyvoltha updates: https://gerrit.opencord.org/#/c/17881/

- reconcile/adapter restart works.  remove forced reboot!
  Needed to create persistence object and recovery procedures
  to fill in variables from omci db and persisted state
  refactored mib_in_sync needed to work with reconcile

- Use new lazy write in-memory storage class.  Drop in replacement for mib_db_dict

- New Twisted etcd storage class.  defers to threads given etcd3 blocks.

- Create function for initializing metrics.  needed for reconcile

- Store onu indication data in etcd so reconciliation can mimic restarting omci

- Check if reconciling to prevent recreation and duplication of ports and provisioning

Change-Id: I08fd5d570059b4ba82e220a20a731dfc3ab1efe1
diff --git a/python/adapters/brcm_openomci_onu/brcm_openomci_onu_adapter.py b/python/adapters/brcm_openomci_onu/brcm_openomci_onu_adapter.py
index 11de8ca..fba1f68 100644
--- a/python/adapters/brcm_openomci_onu/brcm_openomci_onu_adapter.py
+++ b/python/adapters/brcm_openomci_onu/brcm_openomci_onu_adapter.py
@@ -36,6 +36,7 @@
 from pyvoltha.adapters.common.frameio.frameio import hexify
 from pyvoltha.adapters.extensions.omci.openomci_agent import OpenOMCIAgent, OpenOmciAgentDefaults
 from pyvoltha.adapters.extensions.omci.database.mib_db_dict import MibDbVolatileDict
+from pyvoltha.adapters.extensions.omci.database.mib_db_dict_lazy import MibDbLazyWriteDict
 
 from brcm_openomci_onu_handler import BrcmOpenomciOnuHandler
 from omci.brcm_capabilities_task import BrcmCapabilitiesTask
@@ -77,7 +78,7 @@
         self.broadcom_omci = deepcopy(OpenOmciAgentDefaults)
 
         self.broadcom_omci['mib-synchronizer']['audit-delay'] = 0  # disable audits as brcm onu wont upload once provisioned
-        self.broadcom_omci['mib-synchronizer']['database'] = MibDbVolatileDict
+        self.broadcom_omci['mib-synchronizer']['database'] = MibDbLazyWriteDict
         self.broadcom_omci['alarm-synchronizer']['database'] = MibDbVolatileDict
         self.broadcom_omci['omci-capabilities']['tasks']['get-capabilities'] = BrcmCapabilitiesTask
 
@@ -129,8 +130,12 @@
 
     def reconcile_device(self, device):
         self.log.info('reconcile-device', device_id=device.id)
-        self.devices_handlers[device.id] = BrcmOpenomciOnuHandler(self, device.id)
-        reactor.callLater(0, self.devices_handlers[device.id].reconcile, device)
+        if not device.id in self.devices_handlers:
+            self.devices_handlers[device.id] = BrcmOpenomciOnuHandler(self, device.id)
+            reactor.callLater(0, self.devices_handlers[device.id].reconcile, device)
+        else:
+            self.log.debug('already-called-reconcile-device', device_id=device.id)
+        return device
 
     def abandon_device(self, device):
         raise NotImplementedError()