[VOL-4328] Added rw-core restart with ping in background scenario for DT and TT,
[VOL-4367] Added voltha control plane functionality verification after pod restart for DT and TT,
and Also updated cord-robot library requirement to version 2.1.7
Change-Id: Ia72a57ad46fbbf0fa2baac416e591c7ecdc350bf
diff --git a/requirements.txt b/requirements.txt
index 9d78f8e..50fcf46 100755
--- a/requirements.txt
+++ b/requirements.txt
@@ -9,7 +9,7 @@
yamllint==1.23.0
# replace when we can use upstream (needs python 3.6)
git+https://github.com/zdw/robotframework-importresource@b81b87aabaee0594e966687b41e3674b866f28ee
-cord-robot==2.1.6
+cord-robot==2.1.7
virtualenv
matplotlib>=3.0.3
requests==2.24.0
@@ -18,4 +18,4 @@
voltha-protos>=4.2.0
protobuf>=3.15.7
protobuf3-to-dict>=0.1.5
-device-management-interface>=0.13.0
\ No newline at end of file
+device-management-interface>=0.13.0
diff --git a/tests/dt-workflow/Voltha_DT_FailureScenarios.robot b/tests/dt-workflow/Voltha_DT_FailureScenarios.robot
index 85055f2..2b6635b 100755
--- a/tests/dt-workflow/Voltha_DT_FailureScenarios.robot
+++ b/tests/dt-workflow/Voltha_DT_FailureScenarios.robot
@@ -186,7 +186,7 @@
Verify restart openonu-adapter container after subscriber provisioning for DT
[Documentation] Restart openonu-adapter container after VOLTHA is operational.
... Prerequisite : ONUs are authenticated and pingable.
- [Tags] functionalDt Restart-OpenOnu-Dt soak
+ [Tags] Restart-OpenOnu-Dt soak
[Setup] Start Logging Restart-OpenOnu-Dt
[Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
... AND Stop Logging Restart-OpenOnu-Dt
@@ -216,7 +216,7 @@
Verify restart openolt-adapter container after subscriber provisioning for DT
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Prerequisite : ONUs are authenticated and pingable.
- [Tags] functionalDt Restart-OpenOlt-Dt soak
+ [Tags] Restart-OpenOlt-Dt soak
[Setup] Start Logging Restart-OpenOlt-Dt
[Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
... AND Stop Logging Restart-OpenOlt-Dt
@@ -243,20 +243,14 @@
Log to console Pod ${podName} restarted and sanity checks passed successfully
Verify openolt adapter restart before subscriber provisioning for DT
- [Documentation] Deploys an device instance and waits for it to authenticate. After
- ... authentication is successful the rw-core deployment is scaled to 0 instances to
- ... simulate a POD crash. The test then scales the rw-core back to a single instance
- ... and configures ONOS for access. The test succeeds if the device is able to
- ... complete the DHCP sequence.
- [Tags] functionalDt olt-adapter-restart-Dt soak
+ [Documentation] Restart openolt-adapter container before adding the subscriber.
+ [Tags] functionalDt olt-adapter-restart-Dt
[Setup] Start Logging OltAdapterRestart-Dt
- #... AND Clear All Devices Then Create New Device
[Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
... AND Stop Logging OltAdapterRestart-Dt
- # Add OLT and perform sanity test
- #setup
+ # Add OLT device
+ Clear All Devices Then Create New Device
Run Keyword If ${has_dataplane} Clean Up Linux
- #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Set Global Variable ${of_id}
FOR ${I} IN RANGE 0 ${num_all_onus}
@@ -266,9 +260,9 @@
${onu_device_id}= Get Device ID From SN ${src['onu']}
${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
... ${of_id}
- # Bring up the device and verify it authenticates
- Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
- ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed by_dev_id=True
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
+ ... Validate Device ENABLED ACTIVE REACHABLE
+ ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
END
# Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
@@ -302,6 +296,9 @@
Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
... ${onu_port} ${nni_port} ${src['s_tag']}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
+ ... ENABLED ACTIVE REACHABLE
+ ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
@@ -562,11 +559,12 @@
Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
... ${List_ONU_Serial} ${onu_flows}
-Verify restart openonu-adapter container while continuously running ping in background for DT
+Verify restart openonu-adapter container for DT
[Documentation] Restart openonu-adapter container after VOLTHA is operational.
... Run the ping continuously in background during container restart,
... and verify that there should be no affect on the dataplane.
- [Tags] functionalDt RestartOpenOnuPingDt non-critical
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalDt RestartOpenOnuPingDt
[Setup] Start Logging RestartOpenOnuPingDt
[Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
... AND Stop Logging RestartOpenOnuPingDt
@@ -614,12 +612,15 @@
... ${src['container_type']} ${src['container_name']}
Check Ping Result True ${ping_output}
END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart DT
-Verify restart openolt-adapter container while continuously running ping in background for DT
+Verify restart openolt-adapter container for DT
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Run the ping continuously in background during container restart,
... and verify that there should be no affect on the dataplane.
- [Tags] functionalDt RestartOpenOltPingDt non-critical
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalDt RestartOpenOltPingDt
[Setup] Start Logging RestartOpenOltPingDt
[Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
... AND Stop Logging RestartOpenOltPingDt
@@ -667,6 +668,67 @@
... ${src['container_type']} ${src['container_name']}
Check Ping Result True ${ping_output}
END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart DT
+
+Verify restart rw-core container for DT
+ [Documentation] Restart rw-core container after VOLTHA is operational.
+ ... Run the ping continuously in background during container restart,
+ ... and verify that there should be no affect on the dataplane.
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalDt RestartRwCorePingDt
+ [Setup] Start Logging RestartRwCorePingDt
+ [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
+ ... AND Stop Logging RestartRwCorePingDt
+ Clear All Devices Then Create New Device
+ # Performing Sanity Test to make sure subscribers are all DHCP and pingable
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ END
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
+ Log ${podStatusOutput}
+ ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${podName} Set Variable rw-core
+ Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
+ Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
+ ... app ${podName} Running
+ # Wait for 1 min after rw-core is restarted
+ Sleep 60s
+ # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
+ # so restart the port forwarding for the API service
+ Restart VOLTHA Port Forward voltha-api
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
+ Log ${podStatusOutput}
+ ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ END
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
+ ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ Check Ping Result True ${ping_output}
+ END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart DT
*** Keywords ***
Setup Suite
@@ -687,3 +749,45 @@
# Execute normal test Setup Keyword
Setup
+Verify Control Plane After Pod Restart DT
+ [Documentation] Verifies the control plane functionality after the voltha pod restart
+ ... by deleting and re-adding the subscriber
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${of_id}= Get ofID From OLT List ${src['olt']}
+ ${onu_device_id}= Get Device ID From SN ${src['onu']}
+ ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
+ ... ${of_id}
+ # Remove Subscriber Access
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
+ ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ # Disable and Re-Enable the ONU (To replicate DT current workflow)
+ # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
+ Disable Device ${onu_device_id}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
+ ... Validate Device DISABLED UNKNOWN
+ ... REACHABLE ${src['onu']}
+ Enable Device ${onu_device_id}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
+ ... Validate Device ENABLED ACTIVE
+ ... REACHABLE ${src['onu']}
+ # Add Subscriber Access
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
+ ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
+ # Verify subscriber access flows are added for the ONU port
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
+ ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
+ ... ${onu_port} ${nni_port} ${src['s_tag']}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
+ ... Validate Device ENABLED ACTIVE
+ ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ END
diff --git a/tests/tt-workflow/Voltha_TT_FailureScenarios.robot b/tests/tt-workflow/Voltha_TT_FailureScenarios.robot
index ec52e55..f1f425e 100755
--- a/tests/tt-workflow/Voltha_TT_FailureScenarios.robot
+++ b/tests/tt-workflow/Voltha_TT_FailureScenarios.robot
@@ -277,36 +277,6 @@
Run Keyword If ${has_dataplane} Clean Up Linux
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
-Verify restart openonu-adapter container after subscriber provisioning for TT
- [Documentation] Restart openonu-adapter container after VOLTHA is operational.
- ... Prerequisite : ONUs are authenticated and pingable.
- [Tags] functionalTT Restart-OpenOnu-TT
- [Setup] Start Logging Restart-OpenOnu-TT
- [Teardown] Run Keywords Collect Logs
- ... AND Delete All Devices and Verify
- ... AND Collect Logs
- ... AND Stop Logging Restart-OpenOnu-TT
- # Add OLT device
- Setup
- # Performing Sanity Test to make sure subscribers are all DHCP and pingable
- Run Keyword If ${has_dataplane} Clean Up Linux
- Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
- ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
- Log ${podStatusOutput}
- ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
- ${podName} Set Variable adapter-open-onu
- Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
- Sleep 5s
- Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
- ... app ${podName} Running
- Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
- Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
- ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
- Log ${podStatusOutput}
- ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
- Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
- Log to console Pod ${podName} restarted and sanity checks passed successfully
-
Verify restart openolt-adapter container before subscriber provisioning for TT
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
[Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
@@ -343,31 +313,6 @@
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
Log to console Pod ${podName} restarted and sanity checks passed successfully
-Verify restart openolt-adapter container after subscriber provisioning for TT
- [Documentation] Restart openolt-adapter container after VOLTHA is operational.
- [Tags] functionalTT Restart-OpenOlt-TT
- [Setup] Start Logging Restart-OpenOlt-TT
- [Teardown] Run Keywords Collect Logs
- ... AND Stop Logging Restart-OpenOlt-TT
- Setup
- Run Keyword If ${has_dataplane} Clean Up Linux
- Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
- ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
- Log ${podStatusOutput}
- ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
- ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
- Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
- Sleep 5s
- Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
- ... app ${podName} Running
- Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
- Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
- ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
- Log ${podStatusOutput}
- ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
- Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
- Log to console Pod ${podName} restarted and sanity checks passed successfully
-
Verify restart ofagent container after subscriber is provisioned for TT
[Documentation] Restart ofagent container after VOLTHA is operational.
[Tags] functionalTT ofagentRestart-TT
@@ -481,11 +426,12 @@
END
Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
-Verify restart openonu-adapter container while continuously running ping in background for TT
+Verify restart openonu-adapter container for TT
[Documentation] Restart openonu-adapter container after VOLTHA is operational.
... Run the ping continuously in background during container restart,
... and verify that there should be no affect on the dataplane.
- [Tags] functionalTT Restart-OpenOnu-Ping-TT non-critical
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalTT Restart-OpenOnu-Ping-TT
[Setup] Start Logging Restart-OpenOnu-Ping-TT
[Teardown] Run Keywords Collect Logs
... AND Stop Logging Restart-OpenOnu-Ping-TT
@@ -539,12 +485,15 @@
... ${src['container_type']} ${src['container_name']}
Check Ping Result True ${ping_output}
END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart TT
-Verify restart openolt-adapter container while continuously running ping in background for TT
+Verify restart openolt-adapter container for TT
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Run the ping continuously in background during container restart,
... and verify that there should be no affect on the dataplane.
- [Tags] functionalTT Restart-OpenOlt-Ping-TT non-critical
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalTT Restart-OpenOlt-Ping-TT
[Setup] Start Logging Restart-OpenOlt-Ping-TT
[Teardown] Run Keywords Collect Logs
... AND Stop Logging Restart-OpenOlt-Ping-TT
@@ -598,6 +547,73 @@
... ${src['container_type']} ${src['container_name']}
Check Ping Result True ${ping_output}
END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart TT
+
+Verify restart rw-core container for TT
+ [Documentation] Restart rw-core container after VOLTHA is operational.
+ ... Run the ping continuously in background during container restart,
+ ... and verify that there should be no affect on the dataplane.
+ ... Also, verify that the voltha control plane functionality is not affected.
+ [Tags] functionalTT Restart-RwCore-Ping-TT
+ [Setup] Start Logging Restart-RwCore-Ping-TT
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging Restart-RwCore-Ping-TT
+ Clear All Devices Then Create New Device
+ # Performing Sanity Test to make sure subscribers are all DHCP and pingable
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${service_type}= Get Variable Value ${src['service_type']} "null"
+ Continue For Loop If '${service_type}' == 'mcast'
+ ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ END
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
+ Log ${podStatusOutput}
+ ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ ${podName} Set Variable rw-core
+ Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
+ Sleep 5s
+ Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
+ ... app ${podName} Running
+ Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
+ # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
+ # so restart the port forwarding for the API service
+ Restart VOLTHA Port Forward voltha-api
+ ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
+ Log ${podStatusOutput}
+ ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
+ Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${service_type}= Get Variable Value ${src['service_type']} "null"
+ Continue For Loop If '${service_type}' == 'mcast'
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ END
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${service_type}= Get Variable Value ${src['service_type']} "null"
+ Continue For Loop If '${service_type}' == 'mcast'
+ ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
+ ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ Check Ping Result True ${ping_output}
+ END
+ # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
+ Verify Control Plane After Pod Restart TT
*** Keywords ***
Setup Suite
@@ -618,3 +634,35 @@
Delete All Devices and Verify
# Execute normal test Setup Keyword
Setup
+
+Verify Control Plane After Pod Restart TT
+ [Documentation] Verifies the control plane functionality after the voltha pod restart
+ ... by deleting and re-adding the subscriber
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ FOR ${I} IN RANGE 0 ${num_all_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${service_type}= Get Variable Value ${src['service_type']} "null"
+ ${of_id}= Get ofID From OLT List ${src['olt']}
+ ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
+ ... ${of_id} ${src['uni_id']}
+ # Remove Subscriber Access
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
+ ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
+ Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
+ ... Wait Until Keyword Succeeds ${timeout} 2s
+ ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ # Add Subscriber Access
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
+ ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
+ ... Validate Device ENABLED ACTIVE
+ ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
+ Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT one ONU ${src}
+ ... ${dst} ${suppressaddsubscriber}
+ ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
+ ... ${dst} ${suppressaddsubscriber}
+ END