Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 1 | # Copyright 2017 - present Open Networking Foundation |
| 2 | # |
| 3 | # Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | # you may not use this file except in compliance with the License. |
| 5 | # You may obtain a copy of the License at |
| 6 | # |
| 7 | # http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | # |
| 9 | # Unless required by applicable law or agreed to in writing, software |
| 10 | # distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | # See the License for the specific language governing permissions and |
| 13 | # limitations under the License. |
| 14 | # FIXME Can we use the same test against BBSim and Hardware? |
| 15 | |
| 16 | *** Settings *** |
| 17 | Documentation Test various end-to-end scenarios |
| 18 | Suite Setup Setup Suite |
| 19 | Test Setup Setup |
| 20 | Test Teardown Teardown |
| 21 | Suite Teardown Teardown Suite |
| 22 | Library Collections |
| 23 | Library String |
| 24 | Library OperatingSystem |
| 25 | Library XML |
| 26 | Library RequestsLibrary |
| 27 | Library ../../libraries/DependencyLibrary.py |
| 28 | Resource ../../libraries/onos.robot |
| 29 | Resource ../../libraries/voltctl.robot |
| 30 | Resource ../../libraries/voltha.robot |
| 31 | Resource ../../libraries/utils.robot |
| 32 | Resource ../../libraries/k8s.robot |
| 33 | Resource ../../variables/variables.robot |
Andrea Campanella | 15619c7 | 2020-04-30 09:48:04 +0200 | [diff] [blame] | 34 | Resource ../../libraries/power_switch.robot |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 35 | |
| 36 | *** Variables *** |
| 37 | ${POD_NAME} flex-ocp-cord |
| 38 | ${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf |
| 39 | ${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs |
| 40 | #${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf |
| 41 | ${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml |
| 42 | ${HELM_CHARTS_DIR} ~/helm-charts |
| 43 | ${VOLTHA_POD_NUM} 8 |
| 44 | ${NAMESPACE} voltha |
| 45 | # For below variable value, using deployment name as using grep for |
| 46 | # parsing radius pod name, we can also use full radius pod name |
| 47 | ${RESTART_POD_NAME} radius |
| 48 | ${timeout} 60s |
| 49 | ${of_id} 0 |
| 50 | ${logical_id} 0 |
| 51 | ${has_dataplane} True |
| 52 | ${teardown_device} False |
| 53 | ${scripts} ../../scripts |
| 54 | |
| 55 | # Per-test logging on failure is turned off by default; set this variable to enable |
| 56 | ${container_log_dir} ${None} |
| 57 | |
| 58 | *** Test Cases *** |
| 59 | Verify ONU after Rebooting Physically for DT |
| 60 | [Documentation] Test the ONU functionality by physically turning on/off ONU. |
| 61 | ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed) |
| 62 | ... Test case runs only on the PODs that are configured with PowerSwitch that |
| 63 | ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot) |
| 64 | [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 65 | [Setup] Start Logging RebootOnu_PowerSwitch_Dt |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 66 | [Teardown] Run Keywords Collect Logs |
| 67 | ... AND Stop Logging RebootOnu_PowerSwitch_Dt |
| 68 | ... AND Delete Device and Verify |
| 69 | # Add OLT device |
| 70 | Setup |
| 71 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 72 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 73 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 74 | Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password} |
Hardik Windlass | 2eb4718 | 2020-05-19 20:24:12 +0530 | [diff] [blame] | 75 | Run Keyword If ${has_dataplane} Clean Up Linux |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 76 | FOR ${I} IN RANGE 0 ${num_onus} |
| 77 | ${src}= Set Variable ${hosts.src[${I}]} |
| 78 | ${dst}= Set Variable ${hosts.dst[${I}]} |
Hardik Windlass | 0560f80 | 2020-05-06 23:06:53 +0530 | [diff] [blame] | 79 | ${onu_device_id}= Get Device ID From SN ${src['onu']} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 80 | ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s |
| 81 | ... Get ONU Port in ONOS ${src['onu']} ${of_id} |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 82 | # Disable Power Switch |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 83 | Disable Switch Outlet ${src['power_switch_port']} |
| 84 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure |
| 85 | ... Wait Until Keyword Succeeds 60s 2s |
| 86 | ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} |
| 87 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 88 | # Remove Subscriber Access (To replicate DT workflow) |
| 89 | Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP} |
| 90 | ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port} |
Suchitra Vemuri | 1b8d7b8 | 2020-05-29 21:10:12 -0700 | [diff] [blame] | 91 | Sleep 10s |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 92 | # Delete ONU Device (To replicate DT workflow) |
| 93 | Delete Device ${onu_device_id} |
| 94 | Sleep 5s |
| 95 | # Enable Power Switch |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 96 | Enable Switch Outlet ${src['power_switch_port']} |
Hardik Windlass | 40ccffb | 2020-05-11 14:12:07 +0530 | [diff] [blame] | 97 | # Waiting extra time for the ONU to come up |
| 98 | Sleep 60s |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 99 | # Check ONU port is Enabled in ONOS |
| 100 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s |
Suchitra Vemuri | 760bdd3 | 2020-06-17 12:34:48 -0700 | [diff] [blame] | 101 | ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 102 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2 |
| 103 | ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT} |
| 104 | ... volt-add-subscriber-access ${of_id} ${onu_port} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 105 | # Verify ONU state in voltha |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 106 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 107 | ... ENABLED ACTIVE REACHABLE |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 108 | ... ${src['onu']} onu=True onu_reason=omci-flows-pushed |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 109 | # Verify subscriber access flows are added for the ONU port |
| 110 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 111 | ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} |
| 112 | ... ${onu_port} ${nni_port} ${src['s_tag']} |
Hardik Windlass | 2eb4718 | 2020-05-19 20:24:12 +0530 | [diff] [blame] | 113 | #Run Keyword If ${has_dataplane} Clean Up Linux |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 114 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True |
| 115 | ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 116 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
Hardik Windlass | 7c801e6 | 2020-05-04 19:33:21 +0530 | [diff] [blame] | 117 | ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']} |
| 118 | ... ${dst['container_name']} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 119 | Run Keyword And Ignore Error Collect Logs |
| 120 | END |
| 121 | # Verify ONOS Flows |
| 122 | # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream) |
| 123 | ${onos_flows_count}= Evaluate 4 * ${num_onus} |
| 124 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 125 | ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} |
| 126 | ... ${of_id} ${onos_flows_count} |
| 127 | # Verify VOLTHA Flows |
| 128 | # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP |
| 129 | ${olt_flows}= Evaluate 2 * ${num_onus} + 1 |
| 130 | # Number of per ONU Flows equals 2 (one each for downstream and upstream) |
| 131 | ${onu_flows}= Set Variable 2 |
| 132 | Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows} |
| 133 | ${List_ONU_Serial} Create List |
| 134 | Set Suite Variable ${List_ONU_Serial} |
| 135 | Build ONU SN List ${List_ONU_Serial} |
| 136 | Log ${List_ONU_Serial} |
| 137 | Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows |
| 138 | ... ${List_ONU_Serial} ${onu_flows} |
| 139 | |
| 140 | Verify OLT after Rebooting Physically for DT |
| 141 | [Documentation] Test the physical reboot of the OLT |
| 142 | ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed) |
| 143 | ... Test performs a physical reboot, performs "reboot" from the OLT CLI |
| 144 | [Tags] functionalDt PhysicalOltRebootDt VOL-2817 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 145 | [Setup] Start Logging RebootOlt_Physical_Dt |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 146 | [Teardown] Run Keywords Collect Logs |
| 147 | ... AND Stop Logging RebootOlt_Physical_Dt |
| 148 | ... AND Delete Device and Verify |
| 149 | # Add OLT device |
| 150 | Setup |
| 151 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 152 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 153 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 154 | # Reboot the OLT from the OLT CLI |
| 155 | Run Keyword If ${has_dataplane} Login And Run Command On Remote System |
Suchitra Vemuri | 919c2e7 | 2020-07-15 15:23:15 -0700 | [diff] [blame] | 156 | ... sudo reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=# |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 157 | Run Keyword And Ignore Error Collect Logs |
| 158 | FOR ${I} IN RANGE 0 ${num_onus} |
| 159 | ${src}= Set Variable ${hosts.src[${I}]} |
| 160 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 161 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure |
| 162 | ... Wait Until Keyword Succeeds 60s 2s |
| 163 | ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} |
| 164 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 165 | END |
| 166 | # Wait for the OLT to come back up |
Andrea Campanella | 5a4c4b1 | 2020-07-24 23:25:10 +0200 | [diff] [blame^] | 167 | Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 360s 10s |
Suchitra Vemuri | 919c2e7 | 2020-07-15 15:23:15 -0700 | [diff] [blame] | 168 | ... Check Remote System Reachability True ${olt_ssh_ip} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 169 | # Waiting extra time for the ONUs to come up |
| 170 | Sleep 60s |
| 171 | Run Keyword And Ignore Error Collect Logs |
| 172 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 173 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 174 | |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 175 | Verify restart openolt-adapter container after subscriber provisioning for DT |
| 176 | [Documentation] Restart openolt-adapter container after VOLTHA is operational. |
| 177 | ... Prerequisite : ONUs are authenticated and pingable. |
| 178 | [Tags] functionalDt Restart-OpenOlt-Dt |
| 179 | [Setup] Start Logging Restart-OpenOlt-Dt |
| 180 | [Teardown] Run Keywords Collect Logs |
| 181 | ... AND Stop Logging Restart-OpenOlt-Dt |
| 182 | # Add OLT device |
| 183 | setup |
| 184 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 185 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 186 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 187 | ${waitforRestart} Set Variable 120s |
| 188 | ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE} |
| 189 | Log ${podStatusOutput} |
| 190 | ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 191 | ${podName} Set Variable adapter-open-olt |
| 192 | Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName} |
| 193 | Sleep 5s |
| 194 | Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE} |
| 195 | ... app ${podName} Running |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 196 | # Wait for 1min after openolt adapter is restarted |
| 197 | Sleep 60s |
| 198 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 199 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 200 | Run Keyword and Ignore Error Collect Logs |
| 201 | ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE} |
| 202 | Log ${podStatusOutput} |
| 203 | ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l |
| 204 | Should Be Equal As Strings ${countAfterRestart} ${countBforRestart} |
| 205 | Log to console Pod ${podName} restarted and sanity checks passed successfully |
| 206 | |
| 207 | Verify openolt adapter restart before subscriber provisioning for DT |
| 208 | [Documentation] Deploys an device instance and waits for it to authenticate. After |
| 209 | ... authentication is successful the rw-core deployment is scaled to 0 instances to |
| 210 | ... simulate a POD crash. The test then scales the rw-core back to a single instance |
| 211 | ... and configures ONOS for access. The test succeeds if the device is able to |
| 212 | ... complete the DHCP sequence. |
| 213 | [Tags] functionalDt olt-adapter-restart-Dt |
| 214 | [Setup] Start Logging OltAdapterRestart-Dt |
| 215 | #... AND Clear All Devices Then Create New Device |
| 216 | [Teardown] Run Keywords Collect Logs |
| 217 | ... AND Stop Logging OltAdapterRestart-Dt |
| 218 | # Add OLT and perform sanity test |
| 219 | #setup |
| 220 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 221 | #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test |
| 222 | Set Global Variable ${of_id} |
| 223 | |
| 224 | FOR ${I} IN RANGE 0 ${num_onus} |
| 225 | ${src}= Set Variable ${hosts.src[${I}]} |
| 226 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 227 | ${onu_device_id}= Get Device ID From SN ${src['onu']} |
| 228 | ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']} |
| 229 | ... ${of_id} |
| 230 | # Bring up the device and verify it authenticates |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 231 | Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 232 | ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed |
| 233 | END |
| 234 | # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1 |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 235 | Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 0 |
| 236 | Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha open-olt-voltha-adapter-openolt |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 237 | # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 238 | Scale K8s Deployment voltha open-olt-voltha-adapter-openolt 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 239 | Wait Until Keyword Succeeds ${timeout} 2s |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 240 | ... Check Expected Available Deployment Replicas voltha open-olt-voltha-adapter-openolt 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 241 | |
| 242 | # Ensure the device is available in ONOS, this represents system connectivity being restored |
| 243 | Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS |
| 244 | ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id} |
| 245 | |
| 246 | FOR ${I} IN RANGE 0 ${num_onus} |
| 247 | # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly |
| 248 | Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP} |
| 249 | ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port} |
| 250 | # Verify subscriber access flows are added for the ONU port |
| 251 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 252 | ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} |
| 253 | ... ${onu_port} ${nni_port} ${src['s_tag']} |
| 254 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True |
| 255 | ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']} |
| 256 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 257 | ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']} |
| 258 | ... ${dst['container_name']} |
| 259 | END |
| 260 | |
| 261 | Verify restart ofagent container after subscriber is provisioned for DT |
| 262 | [Documentation] Restart ofagent container after VOLTHA is operational. |
| 263 | ... Prerequisite : ONUs are authenticated and pingable. |
| 264 | [Tags] functionalDt ofagentRestart-Dt |
| 265 | [Setup] Start Logging ofagentRestart-Dt |
| 266 | [Teardown] Run Keywords Collect Logs |
| 267 | ... AND Stop Logging ofagentRestart-Dt |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 268 | ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 269 | # set timeout value |
| 270 | ${waitforRestart} Set Variable 120s |
| 271 | ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE} |
| 272 | Log ${podStatusOutput} |
| 273 | ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l |
| 274 | ${podName} Set Variable ofagent |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 275 | Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName} |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 276 | Sleep 60s |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 277 | Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE} |
| 278 | ... app ${podName} Running |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 279 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 280 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 281 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 282 | ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE} |
| 283 | Log ${podStatusOutput} |
| 284 | ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l |
| 285 | Should Be Equal As Strings ${countAfterRestart} ${countBforRestart} |
| 286 | # Scale Down the Of-Agent Deployment |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 287 | Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 288 | Sleep 30s |
| 289 | FOR ${I} IN RANGE 0 ${num_onus} |
| 290 | ${src}= Set Variable ${hosts.src[${I}]} |
| 291 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 292 | Run Keyword and Ignore Error Collect Logs |
| 293 | ${onu_device_id}= Get Device ID From SN ${src['onu']} |
| 294 | ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s |
| 295 | ... Get ONU Port in ONOS ${src['onu']} ${of_id} |
| 296 | # Verify ONU state in voltha |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 297 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 298 | ... ENABLED ACTIVE REACHABLE |
| 299 | ... ${src['onu']} onu=True onu_reason=omci-flows-pushed |
| 300 | # Check ONU port is Disabled in ONOS |
| 301 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s |
Suchitra Vemuri | 760bdd3 | 2020-06-17 12:34:48 -0700 | [diff] [blame] | 302 | ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 303 | # Verify subscriber access flows are added for the ONU port |
| 304 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 305 | ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} |
| 306 | ... ${onu_port} ${nni_port} ${src['s_tag']} |
| 307 | # Verify Ping |
| 308 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True |
| 309 | ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} |
| 310 | ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 311 | Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id} |
| 312 | Run Keyword and Ignore Error Collect Logs |
| 313 | END |
| 314 | # Scale Up the Of-Agent Deployment |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 315 | Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 316 | Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE} |
| 317 | ... Running |
| 318 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 319 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 320 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 321 | Log to console Pod ${podName} restarted and sanity checks passed successfully |
| 322 | |
| 323 | Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT |
| 324 | [Documentation] Deploys an device instance and waits for it to authenticate. After |
| 325 | ... authentication is successful the rw-core deployment is scaled to 0 instances to |
| 326 | ... simulate a POD crash. The test then scales the rw-core back to a single instance |
| 327 | ... and configures ONOS for access. The test succeeds if the device is able to |
| 328 | ... complete the DHCP sequence. |
| 329 | [Tags] functionalDt rwcore-restart-Dt |
| 330 | [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt |
| 331 | ... AND Clear All Devices Then Create New Device |
| 332 | [Teardown] Run Keywords Collect Logs |
| 333 | ... AND Stop Logging RwCoreFailAndRestart-Dt |
| 334 | #... AND Delete Device and Verify |
| 335 | Run Keyword and Ignore Error Collect Logs |
| 336 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 337 | ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number} |
| 338 | Set Global Variable ${of_id} |
| 339 | ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s |
| 340 | ... Get NNI Port in ONOS ${of_id} |
| 341 | Set Global Variable ${nni_port} |
| 342 | FOR ${I} IN RANGE 0 ${num_onus} |
| 343 | ${src}= Set Variable ${hosts.src[${I}]} |
| 344 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 345 | ${onu_device_id}= Get Device ID From SN ${src['onu']} |
| 346 | ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']} |
| 347 | ... ${of_id} |
| 348 | # Bring up the device and verify it authenticates |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 349 | Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 350 | ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded |
| 351 | END |
| 352 | |
| 353 | # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1 |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 354 | Scale K8s Deployment voltha voltha-voltha-rw-core 0 |
| 355 | Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 356 | # Ensure the ofagent POD goes "not-ready" as expected |
| 357 | Wait Until keyword Succeeds ${timeout} 2s |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 358 | ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 359 | # Scale up the core deployment and make sure both it and the ofagent deployment are back |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 360 | Scale K8s Deployment voltha voltha-voltha-rw-core 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 361 | Wait Until Keyword Succeeds ${timeout} 2s |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 362 | ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 363 | Wait Until Keyword Succeeds ${timeout} 2s |
Hardik Windlass | 79b40ff | 2020-06-11 22:55:47 +0530 | [diff] [blame] | 364 | ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1 |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 365 | # For some reason scaling down and up the POD behind a service causes the port forward to stop working, |
| 366 | # so restart the port forwarding for the API service |
David Bainbridge | 07321ff | 2020-06-12 13:55:42 -0700 | [diff] [blame] | 367 | Restart VOLTHA Port Forward voltha-api |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 368 | # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this |
| 369 | # represents system connectivity being restored |
| 370 | Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS |
| 371 | ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id} |
| 372 | |
| 373 | FOR ${I} IN RANGE 0 ${num_onus} |
Andy Bavier | 5209462 | 2020-05-12 15:54:24 -0700 | [diff] [blame] | 374 | ${src}= Set Variable ${hosts.src[${I}]} |
| 375 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 376 | ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']} |
| 377 | ... ${of_id} |
Hardik Windlass | 9df139e | 2020-04-24 14:54:54 +0530 | [diff] [blame] | 378 | # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly |
| 379 | Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP} |
| 380 | ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port} |
| 381 | # Verify subscriber access flows are added for the ONU port |
| 382 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 383 | ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} |
| 384 | ... ${onu_port} ${nni_port} ${src['s_tag']} |
| 385 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True |
| 386 | ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']} |
| 387 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 388 | ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']} |
| 389 | ... ${dst['container_name']} |
| 390 | END |
| 391 | |
Hema | df00368 | 2020-04-28 21:22:22 +0530 | [diff] [blame] | 392 | Verify OLT Soft Reboot for DT |
| 393 | [Documentation] Test soft reboot of the OLT using voltctl command |
| 394 | [Tags] VOL-2818 OLTSoftRebootDt notready |
| 395 | [Setup] Start Logging OLTSoftRebootDt |
| 396 | #... AND Setup |
| 397 | [Teardown] Run Keywords Collect Logs |
| 398 | ... AND Stop Logging OLTSoftRebootDt |
| 399 | #... AND Delete Device and Verify |
| 400 | ## Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 401 | #Run Keyword If ${has_dataplane} Clean Up Linux |
| 402 | #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 403 | # Reboot the OLT using "voltctl device reboot" command |
| 404 | Reboot Device ${olt_device_id} |
| 405 | Run Keyword And Ignore Error Collect Logs |
| 406 | #Verify that ping fails |
| 407 | FOR ${I} IN RANGE 0 ${num_onus} |
| 408 | ${src}= Set Variable ${hosts.src[${I}]} |
| 409 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 410 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure |
| 411 | ... Wait Until Keyword Succeeds 60s 2s |
| 412 | ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} |
| 413 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 414 | END |
| 415 | # Wait for the OLT to come back up |
Andrea Campanella | 5a4c4b1 | 2020-07-24 23:25:10 +0200 | [diff] [blame^] | 416 | Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 360s 10s |
Suchitra Vemuri | 919c2e7 | 2020-07-15 15:23:15 -0700 | [diff] [blame] | 417 | ... Check Remote System Reachability True ${olt_ssh_ip} |
Hema | df00368 | 2020-04-28 21:22:22 +0530 | [diff] [blame] | 418 | # Waiting extra time for the ONUs to come up |
| 419 | Sleep 60s |
| 420 | # Check OLT states |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 421 | Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED ACTIVE REACHABLE |
Hema | df00368 | 2020-04-28 21:22:22 +0530 | [diff] [blame] | 422 | ... ${olt_serial_number} |
| 423 | Run Keyword And Ignore Error Collect Logs |
| 424 | #Check after reboot that ONUs are active, DHCP and pingable |
| 425 | Run Keyword If ${has_dataplane} Clean Up Linux |
| 426 | Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 427 | |
Hema | 8c1a032 | 2020-05-26 16:02:32 +0530 | [diff] [blame] | 428 | Verify ONU Soft Reboot for DT |
| 429 | [Documentation] Test soft reboot of the ONU using voltctl command |
| 430 | [Tags] VOL-2820 ONUSoftRebootDt notready |
| 431 | [Setup] Start Logging ONUSoftRebootDt |
| 432 | #... AND Setup |
| 433 | [Teardown] Run Keywords Collect Logs |
| 434 | ... AND Stop Logging ONUSoftRebootDt |
| 435 | #... AND Delete Device and Verify |
| 436 | # Performing Sanity Test to make sure subscribers are all DHCP and pingable |
| 437 | #Run Keyword If ${has_dataplane} Clean Up Linux |
| 438 | #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT |
| 439 | #Reboot the ONU and verify that ping fails |
| 440 | FOR ${I} IN RANGE 0 ${num_onus} |
| 441 | ${src}= Set Variable ${hosts.src[${I}]} |
| 442 | ${dst}= Set Variable ${hosts.dst[${I}]} |
| 443 | ${onu_device_id}= Get Device ID From SN ${src['onu']} |
| 444 | ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s |
| 445 | ... Get ONU Port in ONOS ${src['onu']} ${of_id} |
| 446 | Reboot Device ${onu_device_id} |
| 447 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure |
| 448 | ... Wait Until Keyword Succeeds 60s 2s |
| 449 | ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} |
| 450 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 451 | Verify ping is succesful except for given device ${num_onus} ${onu_device_id} |
| 452 | # Remove Subscriber Access (To replicate DT workflow) |
| 453 | Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP} |
| 454 | ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port} |
| 455 | # Delete ONU Device (To replicate DT workflow) |
| 456 | Delete Device ${onu_device_id} |
| 457 | Sleep 40s |
| 458 | # Check ONU port is Enabled in ONOS |
| 459 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s |
Suchitra Vemuri | 760bdd3 | 2020-06-17 12:34:48 -0700 | [diff] [blame] | 460 | ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} |
Hema | 8c1a032 | 2020-05-26 16:02:32 +0530 | [diff] [blame] | 461 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2 |
| 462 | ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT} |
| 463 | ... volt-add-subscriber-access ${of_id} ${onu_port} |
| 464 | # Verify ONU state in voltha |
Suchitra Vemuri | ec5da06 | 2020-07-16 22:05:29 -0700 | [diff] [blame] | 465 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device |
Hema | 8c1a032 | 2020-05-26 16:02:32 +0530 | [diff] [blame] | 466 | ... ENABLED ACTIVE REACHABLE |
| 467 | ... ${src['onu']} onu=True onu_reason=omci-flows-pushed |
| 468 | # Verify subscriber access flows are added for the ONU port |
| 469 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 470 | ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} |
| 471 | ... ${onu_port} ${nni_port} ${src['s_tag']} |
| 472 | #Run Keyword If ${has_dataplane} Clean Up Linux |
| 473 | Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True |
| 474 | ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']} |
| 475 | ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']} |
| 476 | ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']} |
| 477 | ... ${dst['container_name']} |
| 478 | Run Keyword And Ignore Error Collect Logs |
| 479 | END |
| 480 | # Verify ONOS Flows |
| 481 | # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream) |
| 482 | ${onos_flows_count}= Evaluate 4 * ${num_onus} |
| 483 | Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s |
| 484 | ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} |
| 485 | ... ${of_id} ${onos_flows_count} |
| 486 | # Verify VOLTHA Flows |
| 487 | # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP |
| 488 | ${olt_flows}= Evaluate 2 * ${num_onus} + 1 |
| 489 | # Number of per ONU Flows equals 2 (one each for downstream and upstream) |
| 490 | ${onu_flows}= Set Variable 2 |
| 491 | Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows} |
| 492 | ${List_ONU_Serial} Create List |
| 493 | Set Suite Variable ${List_ONU_Serial} |
| 494 | Build ONU SN List ${List_ONU_Serial} |
| 495 | Log ${List_ONU_Serial} |
| 496 | Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows |
| 497 | ... ${List_ONU_Serial} ${onu_flows} |
Hema | df00368 | 2020-04-28 21:22:22 +0530 | [diff] [blame] | 498 | |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 499 | *** Keywords *** |
| 500 | Setup Suite |
| 501 | [Documentation] Set up the test suite |
| 502 | Common Test Suite Setup |
Andrea Campanella | 15619c7 | 2020-04-30 09:48:04 +0200 | [diff] [blame] | 503 | #power_switch.robot needs it to support different vendor's power switch |
| 504 | ${switch_type}= Get Variable Value ${web_power_switch.type} |
| 505 | Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type} |
Hardik Windlass | b9bdd50 | 2020-04-17 14:46:21 +0530 | [diff] [blame] | 506 | |
| 507 | Clear All Devices Then Create New Device |
| 508 | [Documentation] Remove any devices from VOLTHA and ONOS |
| 509 | # Remove all devices from voltha and nos |
| 510 | Delete All Devices and Verify |
| 511 | # Execute normal test Setup Keyword |
| 512 | Setup |
| 513 | |