[VOL-4443] OpenOnuAdapter: OLT deletion after adapter restart
Change-Id: I8872185167ab2dc4de8370ea97830d77e8c6bcea
diff --git a/libraries/onu_utilities.robot b/libraries/onu_utilities.robot
index 70476a9..83af853 100755
--- a/libraries/onu_utilities.robot
+++ b/libraries/onu_utilities.robot
@@ -116,7 +116,7 @@
Reconcile Onu Adapter
[Documentation] Restarts the openonu adapter and waits for reconciling is finished and expected oper-state is reached
- [Arguments] ${namespace} ${usekill2restart} ${oper_status}
+ [Arguments] ${namespace} ${usekill2restart} ${oper_status} ${olt_to_be_deleted_sn}=${EMPTY}
# get time of restart of openonu adapter
${restart_ts}= Get Current Date
# restart OpenONU adapter
@@ -126,14 +126,23 @@
${openonu_ready_ts}= Get Pod Ready Timestamp by Label ${namespace} app adapter-open-onu
${restart_duration}= Subtract Date From Date ${openonu_ready_ts} ${restart_ts}
Should Be True ${restart_duration}>0
+ # delete the olt passed, if available (special feature)
+ ${olt_to_be_deleted_device_id}= Run Keyword IF "${olt_to_be_deleted_sn}"!="${EMPTY}"
+ ... Get OLTDeviceID From OLT List ${olt_to_be_deleted_sn}
+ Run Keyword IF "${olt_to_be_deleted_sn}"!="${EMPTY}" Delete Device ${olt_to_be_deleted_device_id}
# wait for the reconcile to complete
# - we check that communication to openonu-adapter is established again
# - we check that all ONUs leave reconcile state by validate a simple voltctl request will not responds with error
Wait Until Keyword Succeeds ${timeout} 1s Validate Last ONU Communication
- Wait Until Keyword Succeeds ${timeout} 1s Validate All Onus Accessible
- # - then we wait that all ONU move to the next state
+ Wait Until Keyword Succeeds ${timeout} 1s Validate All Onus Accessible ${olt_to_be_deleted_sn}
+ # - then we wait that all ONU move to the next state, except ONU belonging to deleted OLT (special feature)
${list_onus} Create List
- Build ONU SN List ${list_onus}
+ FOR ${J} IN RANGE 0 ${num_olts}
+ ${olt_serial_number}= Set Variable ${list_olts}[${J}][sn]
+ Continue For Loop If "${olt_to_be_deleted_sn}"=="${olt_serial_number}"
+ Build ONU SN List ${list_onus} ${olt_serial_number}
+ END
+ Log ${list_onus}
Run Keyword And Ignore Error Wait Until Keyword Succeeds ${timeout}
... 1s Check all ONU OperStatus ${list_onus} ${oper_status}
@@ -141,15 +150,19 @@
[Documentation] This keyword checks all onus accessible (again) with help of a simple voltctl request.
... As long we've got an rc!=0 keyword will fail -> onu is not accessible.
... As get request Onu image list is used, any other get command could be used for this check.
+ ... Will not check ONUs of passed deleted OLT (special feature)
+ [Arguments] ${deleted_olt}=${EMPTY}
${onu_list} Create List
FOR ${I} IN RANGE 0 ${num_all_onus}
${src}= Set Variable ${hosts.src[${I}]}
+ ${olt_serial_number}= Set Variable ${src['olt']}
+ Continue For Loop If "${deleted_olt}"=="${olt_serial_number}"
${onu_device_id}= Get Device ID From SN ${src['onu']}
${onu_id}= Get Index From List ${onu_list} ${onu_device_id}
Continue For Loop If -1 != ${onu_id}
Append To List ${onu_list} ${onu_device_id}
${rc} ${output}= Get Onu Image List ${onu_device_id}
- Should Be True ${rc}==0 Onu ${src['onu']} (${onu_device_id}) still not accessable.
+ Should Be True ${rc}==0 Onu ${src['onu']} (${onu_device_id}) still not accessible.
END
Log Ports
@@ -400,6 +413,17 @@
Append To List ${serialnumberlist} ${serial_number}
END
+Validate Onu Data In Etcd Removed
+ [Documentation] This keyword validates openonu-go-adapter Data stored in etcd are removed.
+ ... In case of a device is passed, only this will be checked.
+ [Arguments] ${namespace}=default ${device_id}=${EMPTY} ${defaultkvstoreprefix}=voltha_voltha
+ ... ${without_pm_data}=True
+ ${kvstoreprefix}= Get Kv Store Prefix ${defaultkvstoreprefix}
+ ${etcddata}= Get ONU Go Adapter ETCD Data ${namespace} ${kvstoreprefix} False ${without_pm_data}
+ ... ${device_id} True
+ Log ${etcddata}
+ Should Be Empty ${etcddata} Stale Openonu Data in Etcd (KV store) ${device_id}
+
Validate Vlan Rules In Etcd
[Documentation] This keyword validates Vlan rules of openonu-go-adapter Data stored in etcd.
... It checks the given number of cookie_slice, match_vid (=4096) and set_vid.
@@ -447,16 +471,20 @@
Get ONU Go Adapter ETCD Data
[Documentation] This keyword delivers openonu-go-adapter Data stored in etcd
[Arguments] ${namespace}=default ${defaultkvstoreprefix}=voltha_voltha ${without_prefix}=True
- ... ${without_pm_data}=True
+ ... ${without_pm_data}=True ${device_id}=${Empty} ${keys_only}=False
${podname}= Set Variable etcd
${kvstoreprefix}= Get Kv Store Prefix ${defaultkvstoreprefix}
${commandget}= Catenate
... /bin/sh -c 'ETCDCTL_API=3 etcdctl get --prefix service/${kvstoreprefix}/openonu'
+ ${commandget}= Run Keyword If ${keys_only} Catenate ${commandget} --keys-only
+ ... ELSE Set Variable ${commandget}
${commandget}= Run Keyword If ${without_prefix} Catenate ${commandget}
... | grep -v service/${kvstoreprefix}/openonu
... ELSE Set Variable ${commandget}
${commandget}= Run Keyword If ${without_pm_data} Catenate ${commandget} | grep -v instances_active
... ELSE Set Variable ${commandget}
+ ${commandget}= Run Keyword If "${device_id}"!="${Empty}" Catenate ${commandget} | grep ${device_id}
+ ... ELSE Set Variable ${commandget}
${result}= Exec Pod In Kube ${namespace} ${podname} ${commandget}
log ${result}
[Return] ${result}
diff --git a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
index 18c6b88..ade42ca 100755
--- a/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
+++ b/tests/openonu-go-adapter/Voltha_ONUReconcileTests.robot
@@ -167,6 +167,30 @@
... AND Teardown Test
... AND Stop Logging ReconcileDisabledOnuDeviceOnuGo
+Olt Deletion After Adapter Restart
+ [Documentation] Validates the OLT deletion after adapter restart
+ ... - prefered environment is two ONUs are active on each of two OLTs, but test works also with single ONU/OLT
+ ... - restart the ONU adapter preferred via "kubectl delete pod"
+ ... - delete one OLT immediately after the restart has been initiated
+ ... - wait until the restart of the ONU adapter and the reconcile processing are finished
+ ... - check whether the ONUs on the remaining OLT have been properly reconciled (if avaialable)
+ ... - check whether the ONUs at the deleted OLT have disappeared from the device list
+ ... - KV store data of these ONUs are deleted under:
+ ... - <kvStorePrefix>/openonu/<deviceId>
+ ... - <kvStorePrefix>/openonu/pm-data/<deviceId>
+ ... - check for not deleted device(s) reason is still the same before restart (if available)
+ ... - delete ONU and MIB-template in KV-store
+ ... Check [VOL-4443] for more details
+ [Tags] functionalOnuGo OltDeletionAfterAdapterRestartOnuGo
+ [Setup] Run Keywords Start Logging OltDeletionAfterAdapterRestartOnuGo
+ ... AND Setup Test
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Do Olt Deletion After Adapter Restart
+ [Teardown] Run Keywords Run Keyword If ${logging} Get Logical Id of OLT
+ ... AND Run Keyword If ${logging} Collect Logs
+ ... AND Teardown Test
+ ... AND Stop Logging OltDeletionAfterAdapterRestartOnuGo
+
*** Keywords ***
Setup Suite
[Documentation] Set up the test suite
@@ -350,3 +374,52 @@
Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT ${suppressaddsubscriber}
... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT ${suppressaddsubscriber}
... ELSE Perform Sanity Test ${suppressaddsubscriber}
+
+
+Do Olt Deletion After Adapter Restart
+ [Documentation] This keyword deletes OLT after adapter restart and checks deleted device(s) and data
+ ... - prefered environment is two ONUs are active on each of two OLTs, but test works also with single ONU/OLT
+ ... - restart the ONU adapter preferred via "kubectl delete pod"
+ ... - delete one OLT immediately after the restart has been initiated
+ ... - wait until the restart of the ONU adapter and the reconcile processing are finished
+ ... - check whether the ONUs on the remaining OLT have been properly reconciled (if avaialable)
+ ... - check whether the ONUs at the deleted OLT have disappeared from the device list
+ ... - KV store data of these ONUs are deleted under:
+ ... - <kvStorePrefix>/openonu/<deviceId>
+ ... - <kvStorePrefix>/openonu/pm-data/<deviceId>
+ ... - check for not deleted device(s) reason is still the same before restart (if available)
+ ... - delete ONU and MIB-template in KV-store
+ ... Check [VOL-4443] for more details
+ FOR ${I} IN RANGE 0 ${num_olts}
+ #get olt serial number
+ ${olt_serial_number}= Set Variable ${list_olts}[${I}][sn]
+ #validate olt states
+ ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
+ Enable Device ${olt_device_id}
+ END
+ # bring all onus to activ -> OMCI-Flows-Pushed
+ Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
+ ... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
+ ... ELSE Perform Sanity Test
+ # OLT#1 will be deleted, get its SN
+ ${olt_to_be_deleted}= Set Variable ${olts[0]['serial']}
+ ${olt_to_be_deleted_device_id}= Get OLTDeviceID From OLT List ${olt_to_be_deleted}
+ # collect all ONU device ids belonging to OLT to be deleted
+ ${onu_device_id_list_should_be_deleted} Create List
+ Build ONU Device Id List ${onu_device_id_list_should_be_deleted} ${olt_to_be_deleted}
+ Log ${onu_device_id_list_should_be_deleted}
+ Reconcile Onu Adapter ${NAMESPACE} ${usekill2restart} ACTIVE ${olt_to_be_deleted}
+ # validate OLT and all corresponding ONUs are removed
+ Validate all ONUS for OLT Removed ${num_all_onus} ${hosts} ${olt_to_be_deleted} ${timeout}
+ Validate Device Removed ${olt_to_be_deleted}
+ # validate for alle removed ONUs KV store date deleted
+ FOR ${onu_device_id} IN @{onu_device_id_list_should_be_deleted}
+ Log ${onu_device_id}
+ Wait Until Keyword Succeeds ${timeout} 1s Validate Onu Data In Etcd Removed ${INFRA_NAMESPACE}
+ ... ${onu_device_id} ${kvstoreprefix} without_pm_data=False
+ END
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ Continue For Loop If "${olt_to_be_deleted}"=="${src['olt']}"
+ Current State Test omci-flows-pushed ${src['onu']}
+ END