[VOL-4443] OpenOnuAdapter: OLT deletion after adapter restart

Change-Id: I8872185167ab2dc4de8370ea97830d77e8c6bcea
diff --git a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
index 18c6b88..ade42ca 100755
--- a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
+++ b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
@@ -167,6 +167,30 @@
     ...    AND    Teardown Test
     ...    AND    Stop Logging    ReconcileDisabledOnuDeviceOnuGo
 
+Olt Deletion After Adapter Restart
+    [Documentation]    Validates the OLT deletion after adapter restart
+    ...    - prefered environment is two ONUs are active on each of two OLTs, but test works also with single ONU/OLT
+    ...    - restart the ONU adapter preferred via "kubectl delete pod"
+    ...    - delete one OLT immediately after the restart has been initiated
+    ...    - wait until the restart of the ONU adapter and the reconcile processing are finished
+    ...    - check whether the ONUs on the remaining OLT have been properly reconciled (if avaialable)
+    ...    - check whether the ONUs at the deleted OLT have disappeared from the device list
+    ...    - KV store data of these ONUs are deleted under:
+    ...        - <kvStorePrefix>/openonu/<deviceId>
+    ...        - <kvStorePrefix>/openonu/pm-data/<deviceId>
+    ...    - check for not deleted device(s) reason is still the same before restart (if available)
+    ...    - delete ONU and MIB-template in KV-store
+    ...    Check [VOL-4443] for more details
+    [Tags]    functionalOnuGo    OltDeletionAfterAdapterRestartOnuGo
+    [Setup]    Run Keywords    Start Logging    OltDeletionAfterAdapterRestartOnuGo
+    ...    AND    Setup Test
+    Run Keyword If    ${has_dataplane}    Clean Up Linux
+    Do Olt Deletion After Adapter Restart
+    [Teardown]    Run Keywords    Run Keyword If    ${logging}    Get Logical Id of OLT
+    ...    AND    Run Keyword If    ${logging}    Collect Logs
+    ...    AND    Teardown Test
+    ...    AND    Stop Logging    OltDeletionAfterAdapterRestartOnuGo
+
 *** Keywords ***
 Setup Suite
     [Documentation]    Set up the test suite
@@ -350,3 +374,52 @@
     Run Keyword If    "${workflow}"=="DT"    Perform Sanity Test DT     ${suppressaddsubscriber}
     ...    ELSE IF    "${workflow}"=="TT"    Perform Sanity Tests TT    ${suppressaddsubscriber}
     ...    ELSE       Perform Sanity Test    ${suppressaddsubscriber}
+
+
+Do Olt Deletion After Adapter Restart
+    [Documentation]    This keyword deletes OLT after adapter restart and checks deleted device(s) and data
+    ...    - prefered environment is two ONUs are active on each of two OLTs, but test works also with single ONU/OLT
+    ...    - restart the ONU adapter preferred via "kubectl delete pod"
+    ...    - delete one OLT immediately after the restart has been initiated
+    ...    - wait until the restart of the ONU adapter and the reconcile processing are finished
+    ...    - check whether the ONUs on the remaining OLT have been properly reconciled (if avaialable)
+    ...    - check whether the ONUs at the deleted OLT have disappeared from the device list
+    ...    - KV store data of these ONUs are deleted under:
+    ...        - <kvStorePrefix>/openonu/<deviceId>
+    ...        - <kvStorePrefix>/openonu/pm-data/<deviceId>
+    ...    - check for not deleted device(s) reason is still the same before restart (if available)
+    ...    - delete ONU and MIB-template in KV-store
+    ...    Check [VOL-4443] for more details
+    FOR    ${I}    IN RANGE    0    ${num_olts}
+        #get olt serial number
+        ${olt_serial_number}=    Set Variable    ${list_olts}[${I}][sn]
+        #validate olt states
+        ${olt_device_id}=    Get OLTDeviceID From OLT List    ${olt_serial_number}
+        Enable Device    ${olt_device_id}
+    END
+    # bring all onus to activ -> OMCI-Flows-Pushed
+    Run Keyword If    "${workflow}"=="DT"    Perform Sanity Test DT
+    ...    ELSE IF    "${workflow}"=="TT"    Perform Sanity Tests TT
+    ...    ELSE       Perform Sanity Test
+    # OLT#1 will be deleted, get its SN
+    ${olt_to_be_deleted}=    Set Variable    ${olts[0]['serial']}
+    ${olt_to_be_deleted_device_id}=    Get OLTDeviceID From OLT List    ${olt_to_be_deleted}
+    # collect all ONU device ids belonging to OLT to be deleted
+    ${onu_device_id_list_should_be_deleted}    Create List
+    Build ONU Device Id List    ${onu_device_id_list_should_be_deleted}    ${olt_to_be_deleted}
+    Log    ${onu_device_id_list_should_be_deleted}
+    Reconcile Onu Adapter    ${NAMESPACE}    ${usekill2restart}    ACTIVE    ${olt_to_be_deleted}
+    # validate OLT and all corresponding ONUs are removed
+    Validate all ONUS for OLT Removed    ${num_all_onus}    ${hosts}    ${olt_to_be_deleted}    ${timeout}
+    Validate Device Removed    ${olt_to_be_deleted}
+    # validate for alle removed ONUs KV store date deleted
+    FOR  ${onu_device_id}  IN  @{onu_device_id_list_should_be_deleted}
+        Log  ${onu_device_id}
+        Wait Until Keyword Succeeds    ${timeout}    1s    Validate Onu Data In Etcd Removed    ${INFRA_NAMESPACE}
+        ...    ${onu_device_id}    ${kvstoreprefix}    without_pm_data=False
+    END
+    FOR    ${I}    IN RANGE    0    ${num_all_onus}
+        ${src}=    Set Variable    ${hosts.src[${I}]}
+        Continue For Loop If    "${olt_to_be_deleted}"=="${src['olt']}"
+        Current State Test    omci-flows-pushed    ${src['onu']}
+    END