[VOL-2670] Reorder failure/restart scenarios, fix issues in the new tests
Change-Id: If378ee3958fbf7b71e8b254d35ef76215dfd6e14
diff --git a/libraries/utils.robot b/libraries/utils.robot
index 8001b1d..8ddbbe9 100644
--- a/libraries/utils.robot
+++ b/libraries/utils.robot
@@ -240,7 +240,7 @@
... Validate OLT Device PREPROVISIONED UNKNOWN UNKNOWN ${EMPTY} ${olt_device_id}
Sleep 5s
Enable Device ${olt_device_id}
- Wait Until Keyword Succeeds ${timeout} 5s
+ Wait Until Keyword Succeeds 180s 5s
... Validate OLT Device ENABLED ACTIVE REACHABLE ${olt_serial_number}
${logical_id}= Get Logical Device ID From SN ${olt_serial_number}
Set Suite Variable ${logical_id}
@@ -253,8 +253,8 @@
Delete Devices In Voltha Root=true
Wait Until Keyword Succeeds ${timeout} 2s Test Empty Device List
# Clear devices from ONOS
- Remove All Devices From ONOS
- ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT}
+ #Remove All Devices From ONOS
+ #... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT}
Teardown
[Documentation] kills processes and cleans up interfaces on src+dst servers
@@ -386,6 +386,15 @@
... ${dst['container_type']} ${dst['container_name']}
END
+Clean WPA Process
+ [Documentation] Kills wpa_supplicant processes only for all RGs
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
+ ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ END
+
Should Be Larger Than
[Documentation] Verify that value_1 is > value_2
[Arguments] ${value_1} ${value_2}
diff --git a/tests/functional/Voltha_ErrorScenarios.robot b/tests/functional/Voltha_ErrorScenarios.robot
index ab53a7b..fcb435e 100644
--- a/tests/functional/Voltha_ErrorScenarios.robot
+++ b/tests/functional/Voltha_ErrorScenarios.robot
@@ -44,7 +44,7 @@
# For below variable value, using deployment name as using grep for
# parsing radius pod name, we can also use full radius pod name
${RESTART_POD_NAME} radius
-${timeout} 90s
+${timeout} 120s
${of_id} 0
${logical_id} 0
${has_dataplane} True
@@ -63,6 +63,9 @@
[Teardown] Run Keywords Collect Logs
... AND Stop Logging AddSameOLT
... AND Announce Message END TEST AddSameOLT
+ # Add OLT device
+ setup
+ # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
Run Keyword If ${has_dataplane} Delete Device and Verify
${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
Set Suite Variable ${olt_device_id}
diff --git a/tests/functional/Voltha_FailureScenarios.robot b/tests/functional/Voltha_FailureScenarios.robot
index ee9850c..5bb7324 100644
--- a/tests/functional/Voltha_FailureScenarios.robot
+++ b/tests/functional/Voltha_FailureScenarios.robot
@@ -107,12 +107,12 @@
Verify restart openolt-adapter container after VOLTHA is operational
[Documentation] Restart openolt-adapter container after VOLTHA is operational.
... Prerequisite : ONUs are authenticated and pingable.
- [Tags] functional VOL-1958 RestartPods released
- [Setup] Run Keywords Announce Message START TEST RestartPods
- ... AND Start Logging RestartPods
+ [Tags] functional VOL-1958 Restart-OpenOlt released
+ [Setup] Run Keywords Announce Message START TEST Restart-OpenOlt
+ ... AND Start Logging Restart-OpenOlt
[Teardown] Run Keywords Collect Logs
- ... AND Stop Logging RestartPods
- ... AND Announce Message END TEST RestartPods
+ ... AND Stop Logging Restart-OpenOlt
+ ... AND Announce Message END TEST Restart-OpenOlt
# Add OLT device
setup
# Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
@@ -134,6 +134,178 @@
Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Log to console Pod ${podName} restarted and sanity checks passed successfully
+Check OLT/ONU Authentication After Radius Pod Restart
+ [Documentation] After radius restart, triggers reassociation, checks status and
+ ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
+ ... wpa supplicant is running in background hence it is recommended to remove
+ ... teardown from previous test or uncomment 'Teardown None'.
+ ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
+ [Tags] functional RadiusRestart released
+ [Setup] Run Keywords Announce Message START TEST RadiusRestart
+ ... AND Start Logging RadiusRestart
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RadiusRestart
+ ... AND Announce Message END TEST RadiusRestart
+ ${waitforRestart} Set Variable 120s
+ Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
+ Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${RESTART_POD_NAME} ${NAMESPACE}
+ ... Running
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${onu_device_id}= Get Device ID From SN ${src['onu']}
+ ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
+ ... Get ONU Port in ONOS ${src['onu']} ${of_id}
+ Wait Until Keyword Succeeds ${timeout} 2s
+ ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Validate DHCP and Ping True True ${src['dp_iface_name']}
+ ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
+ ... ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
+ ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
+ ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
+ Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
+ Run Keyword and Ignore Error Collect Logs
+ END
+
+Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
+ [Documentation] Deploys an device instance and waits for it to authenticate. After
+ ... authentication is successful the rw-core deployment is scaled to 0 instances to
+ ... simulate a POD crash. The test then scales the rw-core back to a single instance
+ ... and configures ONOS for access. The test succeeds if the device is able to
+ ... complete the DHCP sequence.
+ [Tags] functional rwcore-restart
+ [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
+ ... AND Start Logging RwCoreFailAndRestart
+ ... AND Clear All Devices Then Create New Device
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RwCoreFailAndRestart
+ ... AND Announce Message END TEST RwCoreFailAndRestart
+ ... AND Delete Device and Verify
+ Run Keyword and Ignore Error Collect Logs
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
+ Set Global Variable ${of_id}
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${onu_device_id}= Get Device ID From SN ${src['onu']}
+ ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
+ ... ${of_id}
+ # Bring up the device and verify it authenticates
+ Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
+ ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
+ Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
+ ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} ${onu_port}
+ END
+
+ # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
+ Scale K8s Deployment voltha voltha-rw-core 0
+ Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
+ # Ensure the ofagent POD goes "not-ready" as expected
+ Wait Until keyword Succeeds ${timeout} 2s
+ ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
+ # Scale up the core deployment and make sure both it and the ofagent deployment are back
+ Scale K8s Deployment voltha voltha-rw-core 1
+ Wait Until Keyword Succeeds ${timeout} 2s
+ ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
+ Wait Until Keyword Succeeds ${timeout} 2s
+ ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
+ # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
+ # so restart the port forwarding for the API service
+ Restart VOLTHA Port Foward voltha-api-minimal
+ # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
+ # represents system connectivity being restored
+ Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
+ ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
+
+ FOR ${I} IN RANGE 0 ${num_onus}
+ # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
+ ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
+ ... ${dst['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
+ ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
+ END
+
+Sanity E2E Test for OLT/ONU on POD With OLT Adapters Fail and Restart
+ [Documentation] Deploys an device instance and waits for it to authenticate. After
+ ... authentication is successful the rw-core deployment is scaled to 0 instances to
+ ... simulate a POD crash. The test then scales the rw-core back to a single instance
+ ... and configures ONOS for access. The test succeeds if the device is able to
+ ... complete the DHCP sequence.
+ [Tags] bbsim olt-adapter-restart
+ [Setup] Run Keywords Announce Message START TEST OltAdapterRestart
+ ... AND Start Logging OltAdapterRestart
+ ... AND Clear All Devices Then Create New Device
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging OltAdapterRestart
+ ... AND Announce Message END TEST OltAdapterRestart
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
+ Set Global Variable ${of_id}
+
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${onu_device_id}= Get Device ID From SN ${src['onu']}
+ ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
+ ... ${of_id}
+
+ # Bring up the device and verify it authenticates
+ Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
+ ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
+ Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
+ ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
+ ... ${src['container_type']} ${src['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} ${onu_port}
+ END
+ # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
+ Scale K8s Deployment voltha adapter-open-olt 0
+ Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
+ # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
+ Scale K8s Deployment voltha adapter-open-olt 1
+ Wait Until Keyword Succeeds ${timeout} 2s
+ ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
+
+ # Ensure the device is available in ONOS, this represents system connectivity being restored
+ Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
+ ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
+
+ FOR ${I} IN RANGE 0 ${num_onus}
+ # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
+ Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
+ ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
+ ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
+ ... ${dst['container_name']}
+ Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
+ ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
+ END
+
Verify restart ofagent container after VOLTHA is operational
[Documentation] Restart ofagent container after VOLTHA is operational.
... Please note this test case should be run before the restart of other containers.
@@ -243,3 +415,15 @@
#Run Keyword If ${has_dataplane} Clean Up Linux
#Check after reboot that ONUs are active, authenticated/DHCP/pingable
#Perform Sanity Test
+
+*** Keywords ***
+Setup Suite
+ [Documentation] Set up the test suite
+ Common Test Suite Setup
+
+Clear All Devices Then Create New Device
+ [Documentation] Remove any devices from VOLTHA and ONOS
+ # Remove all devices from voltha and nos
+ Delete All Devices and Verify
+ # Execute normal test Setup Keyword
+ Setup
diff --git a/tests/functional/Voltha_PODTests.robot b/tests/functional/Voltha_PODTests.robot
index e26f820..1d172a5 100644
--- a/tests/functional/Voltha_PODTests.robot
+++ b/tests/functional/Voltha_PODTests.robot
@@ -190,49 +190,6 @@
Run Keyword and Ignore Error Collect Logs
END
-Check OLT/ONU Authentication After Radius Pod Restart
- [Documentation] After radius restart, triggers reassociation, checks status and
- ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
- ... wpa supplicant is running in background hence it is recommended to remove
- ... teardown from previous test or uncomment 'Teardown None'.
- ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
- [Tags] functional RadiusRestart released
- [Setup] Run Keywords Announce Message START TEST RadiusRestart
- ... AND Start Logging RadiusRestart
- [Teardown] Run Keywords Collect Logs
- ... AND Stop Logging RadiusRestart
- ... AND Announce Message END TEST RadiusRestart
- ${waitforRestart} Set Variable 120s
- Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
- Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${RESTART_POD_NAME} ${NAMESPACE}
- ... Running
- FOR ${I} IN RANGE 0 ${num_onus}
- ${src}= Set Variable ${hosts.src[${I}]}
- ${dst}= Set Variable ${hosts.dst[${I}]}
- ${onu_device_id}= Get Device ID From SN ${src['onu']}
- ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
- ... Get ONU Port in ONOS ${src['onu']} ${of_id}
- Wait Until Keyword Succeeds ${timeout} 2s
- ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
- ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
- ... ${src['ip']} ${src['user']} ${src['pass']}
- ... ${src['container_type']} ${src['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
- ... Validate DHCP and Ping True True ${src['dp_iface_name']}
- ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
- ... ${src['ip']} ${src['user']} ${src['pass']}
- ... ${src['container_type']} ${src['container_name']}
- ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
- ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
- ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
- Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
- Run Keyword and Ignore Error Collect Logs
- END
-
Check DHCP attempt fails when subscriber is not added
[Documentation] Validates when removed subscriber access, DHCP attempt, ping fails and
... when again added subscriber access, DHCP attempt, ping succeeds
@@ -402,167 +359,51 @@
${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${olt_device_id}
Should Be Equal As Integers ${rc} 0
Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Test Empty Device List
- #Adding setup here to add the devices back since this TC removes the devices
- Run Keyword If ${has_dataplane} sleep 180s
- #setup
-
-Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
- [Documentation] Deploys an device instance and waits for it to authenticate. After
- ... authentication is successful the rw-core deployment is scaled to 0 instances to
- ... simulate a POD crash. The test then scales the rw-core back to a single instance
- ... and configures ONOS for access. The test succeeds if the device is able to
- ... complete the DHCP sequence.
- [Tags] bbsim rwcore-restart
- [Setup] Run Keywords Announce Message START TEST RwCoreFailAndRestart
- ... AND Start Logging RwCoreFailAndRestart
- ... AND Clear All Devices Then Create New Device
- [Teardown] Run Keywords Collect Logs
- ... AND Stop Logging RwCoreFailAndRestart
- ... AND Announce Message END TEST RwCoreFailAndRestart
- ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
- Set Global Variable ${of_id}
- FOR ${I} IN RANGE 0 ${num_onus}
- ${src}= Set Variable ${hosts.src[${I}]}
- ${dst}= Set Variable ${hosts.dst[${I}]}
- ${onu_device_id}= Get Device ID From SN ${src['onu']}
- ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
- ... ${of_id}
- # Bring up the device and verify it authenticates
- Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
- ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
- Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
- ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
- ... ${src['container_type']} ${src['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} ${onu_port}
- END
-
- # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
- Scale K8s Deployment voltha voltha-rw-core 0
- Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
- # Ensure the ofagent POD goes "not-ready" as expected
- Wait Until keyword Succeeds ${timeout} 2s
- ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
- # Scale up the core deployment and make sure both it and the ofagent deployment are back
- Scale K8s Deployment voltha voltha-rw-core 1
- Wait Until Keyword Succeeds ${timeout} 2s
- ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
- Wait Until Keyword Succeeds ${timeout} 2s
- ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
- # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
- # so restart the port forwarding for the API service
- Restart VOLTHA Port Foward voltha-api-minimal
- # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
- # represents system connectivity being restored
- Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
- ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
-
- FOR ${I} IN RANGE 0 ${num_onus}
- # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
- Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
- ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
- ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
- ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
- ... ${dst['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
- ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
- END
-
-Sanity E2E Test for OLT/ONU on POD With OLT Adapters Fail and Restart
- [Documentation] Deploys an device instance and waits for it to authenticate. After
- ... authentication is successful the rw-core deployment is scaled to 0 instances to
- ... simulate a POD crash. The test then scales the rw-core back to a single instance
- ... and configures ONOS for access. The test succeeds if the device is able to
- ... complete the DHCP sequence.
- [Tags] bbsim olt-adapter-restart
- [Setup] Run Keywords Announce Message START TEST OltAdapterRestart
- ... AND Start Logging OltAdapterRestart
- ... AND Clear All Devices Then Create New Device
- [Teardown] Run Keywords Collect Logs
- ... AND Stop Logging OltAdapterRestart
- ... AND Announce Message END TEST OltAdapterRestart
- ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
- Set Global Variable ${of_id}
-
- FOR ${I} IN RANGE 0 ${num_onus}
- ${src}= Set Variable ${hosts.src[${I}]}
- ${dst}= Set Variable ${hosts.dst[${I}]}
- ${onu_device_id}= Get Device ID From SN ${src['onu']}
- ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
- ... ${of_id}
-
- # Bring up the device and verify it authenticates
- Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
- ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
- Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
- ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
- ... ${src['container_type']} ${src['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} ${onu_port}
- END
-
- # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
- Scale K8s Deployment voltha adapter-open-olt 0
- Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha adapter-open-olt
- # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
- Scale K8s Deployment voltha adapter-open-olt 1
- Wait Until Keyword Succeeds ${timeout} 2s
- ... Check Expected Available Deployment Replicas voltha adapter-open-olt 1
-
- # Ensure the device is available in ONOS, this represents system connectivity being restored
- Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
- ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
-
- FOR ${I} IN RANGE 0 ${num_onus}
- # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
- Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
- ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
- Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
- ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
- ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
- ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
- ... ${dst['container_name']}
- Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
- ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
- END
Test Disable and Enable ONU and check authentication
[Documentation] Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
... Perform disable on the ONUs and validate that the authentication do not succeed
... Perform enable on the ONUs and validate that authentication successful
- [Tags] functional DisableEnableONU_AuthCheck notready
- [Setup] None
- [Teardown] None
+ [Tags] functional DisableONU_AuthCheck
+ # Creates Devices in the Setup
+ [Setup] Run Keywords Announce Message START TEST DisableDeleteONUandOLT
+ ... AND Start Logging DisableONU_AuthCheck
+ ... AND Setup
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging DisableONU_AuthCheck
+ ... AND Announce Message END TEST DisableONU_AuthCheck
+ ... AND Delete Device and Verify
+ Run Keyword and Ignore Error Collect Logs
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
+ Clean WPA Process
FOR ${I} IN RANGE 0 ${num_onus}
${src}= Set Variable ${hosts.src[${I}]}
${dst}= Set Variable ${hosts.dst[${I}]}
+ Run Keyword and Ignore Error Collect Logs
${onu_device_id}= Get Device ID From SN ${src['onu']}
${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
... ${of_id}
Disable Device ${onu_device_id}
Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED UNKNOWN
... REACHABLE ${src['onu']} onu=false
- Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds ${timeout} 2s
- ... Validate Authentication After Reassociate False ${src['dp_iface_name']}
- ... ${src['ip']} ${src['user']} ${src['pass']}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication False
+ ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
... ${src['container_type']} ${src['container_name']}
- Enable Device ${onu_device_id}
- Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
- ... ENABLED ACTIVE REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
- Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds ${timeout} 2s
- ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
- ... ${src['ip']} ${src['user']} ${src['pass']}
+ Enable Device ${onu_device_id}
+ Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
+ ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ Run Keyword and Ignore Error Collect Logs
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
+ ... ENABLED ACTIVE REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
+ ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
... ${src['container_type']} ${src['container_name']}
Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
END
Run Keyword and Ignore Error Collect Logs
+
*** Keywords ***
Setup Suite
[Documentation] Set up the test suite