[VOL-2817] [VOL-2819]: DT Workflow: ONU/OLT Hard Reboot TC
Change-Id: Ia1f12380d41db5713e8f5b00804ed4a079b6008a
diff --git a/tests/dt-workflow/Voltha_DT_FailureScenarios.robot b/tests/dt-workflow/Voltha_DT_FailureScenarios.robot
new file mode 100644
index 0000000..1326304
--- /dev/null
+++ b/tests/dt-workflow/Voltha_DT_FailureScenarios.robot
@@ -0,0 +1,168 @@
+# Copyright 2017 - present Open Networking Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# FIXME Can we use the same test against BBSim and Hardware?
+
+*** Settings ***
+Documentation Test various end-to-end scenarios
+Suite Setup Setup Suite
+Test Setup Setup
+Test Teardown Teardown
+Suite Teardown Teardown Suite
+Library Collections
+Library String
+Library OperatingSystem
+Library XML
+Library RequestsLibrary
+Library ../../libraries/DependencyLibrary.py
+Resource ../../libraries/onos.robot
+Resource ../../libraries/voltctl.robot
+Resource ../../libraries/voltha.robot
+Resource ../../libraries/utils.robot
+Resource ../../libraries/k8s.robot
+Resource ../../variables/variables.robot
+
+*** Variables ***
+${POD_NAME} flex-ocp-cord
+${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
+${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
+#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
+${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
+${HELM_CHARTS_DIR} ~/helm-charts
+${VOLTHA_POD_NUM} 8
+${NAMESPACE} voltha
+# For below variable value, using deployment name as using grep for
+# parsing radius pod name, we can also use full radius pod name
+${RESTART_POD_NAME} radius
+${timeout} 60s
+${of_id} 0
+${logical_id} 0
+${has_dataplane} True
+${teardown_device} False
+${scripts} ../../scripts
+
+# Per-test logging on failure is turned off by default; set this variable to enable
+${container_log_dir} ${None}
+
+*** Test Cases ***
+Verify ONU after Rebooting Physically for DT
+ [Documentation] Test the ONU functionality by physically turning on/off ONU.
+ ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
+ ... Test case runs only on the PODs that are configured with PowerSwitch that
+ ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
+ [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
+ [Setup] Run Keywords Start Logging RebootOnu_PowerSwitch_Dt
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RebootOnu_PowerSwitch_Dt
+ ... AND Delete Device and Verify
+ # Add OLT device
+ Setup
+ # Performing Sanity Test to make sure subscribers are all DHCP and pingable
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
+ Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
+ ... Get ONU Port in ONOS ${src['onu']} ${of_id}
+ Disable Switch Outlet ${src['power_switch_port']}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+
+ Enable Switch Outlet ${src['power_switch_port']}
+ # Check ONU port is Enabled in ONOS
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
+ ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
+ # Verify ONU state in voltha
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
+ ... ENABLED ACTIVE REACHABLE
+ ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
+ # Verify subscriber access flows are added for the ONU port
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
+ ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
+ ... ${onu_port} ${nni_port} ${src['s_tag']}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ Run Keyword And Ignore Error Collect Logs
+ END
+ # Verify ONOS Flows
+ # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
+ ${onos_flows_count}= Evaluate 4 * ${num_onus}
+ Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
+ ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
+ ... ${of_id} ${onos_flows_count}
+ # Verify VOLTHA Flows
+ # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
+ ${olt_flows}= Evaluate 2 * ${num_onus} + 1
+ # Number of per ONU Flows equals 2 (one each for downstream and upstream)
+ ${onu_flows}= Set Variable 2
+ Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
+ ${List_ONU_Serial} Create List
+ Set Suite Variable ${List_ONU_Serial}
+ Build ONU SN List ${List_ONU_Serial}
+ Log ${List_ONU_Serial}
+ Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
+ ... ${List_ONU_Serial} ${onu_flows}
+
+Verify OLT after Rebooting Physically for DT
+ [Documentation] Test the physical reboot of the OLT
+ ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
+ ... Test performs a physical reboot, performs "reboot" from the OLT CLI
+ [Tags] functionalDt PhysicalOltRebootDt VOL-2817
+ [Setup] Run Keywords Start Logging RebootOlt_Physical_Dt
+ [Teardown] Run Keywords Collect Logs
+ ... AND Stop Logging RebootOlt_Physical_Dt
+ ... AND Delete Device and Verify
+ # Add OLT device
+ Setup
+ # Performing Sanity Test to make sure subscribers are all DHCP and pingable
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
+ # Reboot the OLT from the OLT CLI
+ Run Keyword If ${has_dataplane} Login And Run Command On Remote System
+ ... sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
+ Run Keyword And Ignore Error Collect Logs
+ FOR ${I} IN RANGE 0 ${num_onus}
+ ${src}= Set Variable ${hosts.src[${I}]}
+ ${dst}= Set Variable ${hosts.dst[${I}]}
+ Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
+ ... Wait Until Keyword Succeeds 60s 2s
+ ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
+ ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
+ END
+ # Wait for the OLT to come back up
+ Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
+ ... Check Remote System Reachability True ${olt_ip}
+ # Waiting extra time for the ONUs to come up
+ Sleep 60s
+ Run Keyword And Ignore Error Collect Logs
+ Run Keyword If ${has_dataplane} Clean Up Linux
+ Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
+
+*** Keywords ***
+Setup Suite
+ [Documentation] Set up the test suite
+ Common Test Suite Setup
+
+Clear All Devices Then Create New Device
+ [Documentation] Remove any devices from VOLTHA and ONOS
+ # Remove all devices from voltha and nos
+ Delete All Devices and Verify
+ # Execute normal test Setup Keyword
+ Setup
+