Suchitra Vemuri | f8ef284 | 2018-12-18 15:11:39 -0800 | [diff] [blame] | 1 | # Copyright 2017-present Open Networking Foundation |
| 2 | # |
| 3 | # Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | # you may not use this file except in compliance with the License. |
| 5 | # You may obtain a copy of the License at |
| 6 | # |
| 7 | # http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | # |
| 9 | # Unless required by applicable law or agreed to in writing, software |
| 10 | # distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | # See the License for the specific language governing permissions and |
| 13 | # limitations under the License. |
| 14 | |
| 15 | *** Settings *** |
| 16 | Documentation Test various data plane failure scenarios with ATT workflow |
| 17 | Suite Setup Setup Suite |
| 18 | Suite Teardown Teardown Suite |
| 19 | Test Setup Setup Test |
| 20 | Test Teardown Teardown Test |
| 21 | Library Collections |
| 22 | Library String |
| 23 | Library OperatingSystem |
| 24 | Library XML |
| 25 | Library RequestsLibrary |
| 26 | Library ../../Framework/utils/utils.py |
| 27 | Resource ../../Framework/utils/utils.robot |
| 28 | Library ../../Framework/restApi.py |
| 29 | Resource ../../Framework/Subscriber.robot |
| 30 | Resource ../../Framework/ATTWorkFlowDriver.robot |
| 31 | Resource ../../Framework/Kubernetes.robot |
| 32 | Resource ../../Framework/ONU.robot |
| 33 | Resource ../../Framework/OLT.robot |
| 34 | Resource ../../Framework/DHCP.robot |
| 35 | Variables ../../Properties/RestApiProperties.py |
| 36 | |
| 37 | *** Variables *** |
| 38 | ${POD_NAME} flex-pod1-olt |
| 39 | ${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs |
| 40 | ${HELM_CHARTS_DIR} ~/helm-charts |
| 41 | ${WHITELIST_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTWhiteList.json |
| 42 | ${SUBSCRIBER_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTSubscriber.json |
| 43 | ${VOLT_DEVICE_PATHFILE} ${CURDIR}/data/${POD_NAME}/RealOLTDevice.json |
| 44 | ${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf |
| 45 | ${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml |
| 46 | ${VOLTHA_POD_NUM} 8 |
| 47 | |
| 48 | *** Test Cases *** |
| 49 | |
| 50 | OLT Reboot |
| 51 | [Documentation] Validates E2E Ping Connectivity and object states for the given scenario: |
| 52 | ... Configure whitelist with correct ONU location |
| 53 | ... Validate successful authentication/DHCP/E2E ping |
| 54 | ... Reboots OLT |
| 55 | ... Validate that pings fail |
| 56 | ... Validate successful authentication/DHCP/E2E ping after OLT comes back up |
| 57 | [Tags] olt1 |
| 58 | Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device} |
| 59 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device} |
| 60 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device} |
| 61 | Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 62 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 63 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 64 | Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 65 | # Reboot OLT |
| 66 | Login And Run Command On Remote System sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=# |
| 67 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip} |
Suchitra Vemuri | 6fd2d12 | 2019-05-13 14:40:32 -0700 | [diff] [blame] | 68 | Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
Suchitra Vemuri | f8ef284 | 2018-12-18 15:11:39 -0800 | [diff] [blame] | 69 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip} |
| 70 | Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass} |
| 71 | # Validate successful pings since the OLT is Up |
| 72 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 73 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 74 | Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} |
| 75 | |
Suchitra Vemuri | 197fbf8 | 2019-01-03 14:06:42 -0800 | [diff] [blame] | 76 | Fabric Switch Reboot |
| 77 | [Documentation] Validates E2E Ping Connectivity and object states for the given scenario: |
| 78 | ... Configure whitelist with correct ONU location |
| 79 | ... Validate successful authentication/DHCP/E2E ping |
| 80 | ... Reboots Fabric Switch |
| 81 | ... Validate that pings fail |
| 82 | ... Validate successful authentication/DHCP/E2E ping after OLT comes back up |
| 83 | #[Setup] None |
Suchitra Vemuri | 64d5c86 | 2019-01-07 19:09:11 -0800 | [diff] [blame] | 84 | #[Teardown] None |
Suchitra Vemuri | 197fbf8 | 2019-01-03 14:06:42 -0800 | [diff] [blame] | 85 | [Tags] fabric1 |
| 86 | Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device} |
| 87 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device} |
| 88 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device} |
| 89 | Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 90 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 91 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 92 | Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 93 | # Reboot Fabric Switch |
| 94 | Login And Run Command On Remote System sudo reboot ${fabric_ip} ${fabric_user} ${fabric_pass} prompt=# |
| 95 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip} |
| 96 | Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} |
| 97 | #Validate DHCP and Ping False False ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 98 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${fabric_ip} |
| 99 | # Validate successful pings since Fabric Switch is Up |
| 100 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 101 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 102 | Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} |
Suchitra Vemuri | f8ef284 | 2018-12-18 15:11:39 -0800 | [diff] [blame] | 103 | |
Suchitra Vemuri | 64d5c86 | 2019-01-07 19:09:11 -0800 | [diff] [blame] | 104 | Subscriber(RG) Reboot |
| 105 | [Documentation] Validates E2E Ping Connectivity and object states for the given scenario: |
| 106 | ... Configure whitelist with correct ONU location |
| 107 | ... Validate successful authentication/DHCP/E2E ping |
| 108 | ... Reboots RG |
| 109 | ... Validate that pings fail |
| 110 | ... Validate successful authentication/DHCP/E2E ping after OLT comes back up |
| 111 | [Tags] RG-reboot |
| 112 | Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device} |
| 113 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device} |
| 114 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device} |
| 115 | Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 116 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 117 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 118 | Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 119 | # Reboot RG |
| 120 | Login And Run Command On Remote System sudo reboot ${src_ip} ${src_user} ${src_pass} prompt=$ |
| 121 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip} |
| 122 | Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} |
| 123 | Wait Until Keyword Succeeds 150s 10s Check Remote System Reachability True ${src_ip} |
| 124 | Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass} |
| 125 | # Perform Reauthentication/DHCP and Ping |
| 126 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device} |
| 127 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device} |
| 128 | Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 129 | Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device} |
| 130 | Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device} |
| 131 | Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 132 | |
Suchitra Vemuri | f8ef284 | 2018-12-18 15:11:39 -0800 | [diff] [blame] | 133 | *** Keywords *** |
| 134 | Setup Suite |
| 135 | ${auth} = Create List ${XOS_USER} ${XOS_PASSWD} |
| 136 | ${HEADERS} Create Dictionary Content-Type=application/json |
| 137 | Create Session ${server_ip} http://${server_ip}:${server_port} auth=${AUTH} headers=${HEADERS} |
| 138 | ${att_workflow_service_id}= Get Service Owner Id ${ATT_SERVICE} |
| 139 | ${volt_service_id}= Get Service Owner Id ${VOLT_SERVICE} |
| 140 | ${AttWhiteListList}= utils.jsonToList ${WHITELIST_PATHFILE} AttWhiteListInfo |
| 141 | Set Suite Variable ${AttWhiteListList} |
| 142 | ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0 |
| 143 | ${AttWhiteListDict}= utils.setFieldValueInDict ${AttWhiteListDict} owner_id ${att_workflow_service_id} |
| 144 | ${onu_device}= Get From Dictionary ${AttWhiteListDict} serial_number |
| 145 | Set Global Variable ${onu_device} |
| 146 | ${onu_location}= Get From Dictionary ${AttWhiteListDict} pon_port_id |
| 147 | Set Global Variable ${onu_location} |
| 148 | ${SubscriberList}= utils.jsonToList ${SUBSCRIBER_PATHFILE} SubscriberInfo |
| 149 | Set Global Variable ${SubscriberList} |
| 150 | ${SubscriberDict}= utils.listToDict ${SubscriberList} 0 |
| 151 | ${s_tag}= utils.getFieldValueFromDict ${SubscriberDict} s_tag |
| 152 | ${c_tag}= utils.getFieldValueFromDict ${SubscriberDict} c_tag |
| 153 | ${VoltDeviceList}= utils.jsonToList ${VOLT_DEVICE_PATHFILE} VOLTDeviceInfo |
| 154 | Set Global Variable ${VoltDeviceList} |
| 155 | Set Suite Variable ${s_tag} |
| 156 | Set Suite Variable ${c_tag} |
| 157 | Set Global Variable ${export_kubeconfig} export KUBECONFIG=${KUBERNETES_CONF} |
| 158 | # Read variables from yaml file |
| 159 | ${src_ip}= Evaluate ${hosts}.get("src").get("ip") |
| 160 | ${src_user}= Evaluate ${hosts}.get("src").get("user") |
| 161 | ${src_pass}= Evaluate ${hosts}.get("src").get("pass") |
| 162 | ${src_container_type}= Evaluate ${hosts}.get("src").get("container_type") |
| 163 | ${src_container_name}= Evaluate ${hosts}.get("src").get("container_name") |
| 164 | ${src_iface}= Evaluate ${hosts}.get("src").get("dp_iface_name") |
| 165 | ${dst_ip}= Evaluate ${hosts}.get("dst").get("ip") |
| 166 | ${dst_user} = Evaluate ${hosts}.get("dst").get("user") |
| 167 | ${dst_pass}= Evaluate ${hosts}.get("dst").get("pass") |
| 168 | ${dst_container_type}= Evaluate ${hosts}.get("dst").get("container_type") |
| 169 | ${dst_container_name}= Evaluate ${hosts}.get("dst").get("container_name") |
| 170 | ${dst_dp_iface}= Evaluate ${hosts}.get("dst").get("dp_iface_name") |
| 171 | ${dst_dp_ip}= Evaluate ${hosts}.get("dst").get("dp_iface_ip_qinq") |
| 172 | ${olt_ip}= Evaluate ${olts}[0].get("ip") |
| 173 | ${olt_user}= Evaluate ${olts}[0].get("user") |
| 174 | ${olt_pass}= Evaluate ${olts}[0].get("pass") |
| 175 | ${fabric_ip}= Evaluate ${fabric_switches}[0].get("ip") |
| 176 | ${fabric_user}= Evaluate ${fabric_switches}[0].get("user") |
| 177 | ${fabric_pass}= Evaluate ${fabric_switches}[0].get("pass") |
| 178 | ${k8s_node_ip}= Evaluate ${nodes}[0].get("ip") |
| 179 | ${k8s_node_user}= Evaluate ${nodes}[0].get("user") |
| 180 | ${k8s_node_pass}= Evaluate ${nodes}[0].get("pass") |
| 181 | Set Suite Variable ${src_ip} |
| 182 | Set Suite Variable ${src_user} |
| 183 | Set Suite Variable ${src_pass} |
| 184 | Set Suite Variable ${src_container_type} |
| 185 | Set Suite Variable ${src_container_name} |
| 186 | Set Suite Variable ${src_iface} |
| 187 | Set Suite Variable ${dst_ip} |
| 188 | Set Suite Variable ${dst_user} |
| 189 | Set Suite Variable ${dst_pass} |
| 190 | Set Suite Variable ${dst_container_type} |
| 191 | Set Suite Variable ${dst_container_name} |
| 192 | Set Suite Variable ${dst_dp_iface} |
| 193 | Set Suite Variable ${dst_dp_ip} |
| 194 | Set Suite Variable ${olt_ip} |
| 195 | Set Suite Variable ${olt_user} |
| 196 | Set Suite Variable ${olt_pass} |
| 197 | Set Suite Variable ${fabric_ip} |
| 198 | Set Suite Variable ${fabric_user} |
| 199 | Set Suite Variable ${fabric_pass} |
| 200 | Set Suite Variable ${k8s_node_ip} |
| 201 | Set Suite Variable ${k8s_node_user} |
| 202 | Set Suite Variable ${k8s_node_pass} |
| 203 | @{container_list}= Create List |
| 204 | Append To List ${container_list} att-workflow-att-workflow-driver |
| 205 | Append To List ${container_list} att-workflow-volt |
| 206 | Append To List ${container_list} onos |
| 207 | Append To List ${container_list} xos-core |
| 208 | Append To List ${container_list} vcore |
| 209 | Set Suite Variable ${container_list} |
| 210 | ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass} |
| 211 | Set Suite Variable ${datetime} |
| 212 | |
| 213 | Teardown Suite |
| 214 | [Documentation] Performs any additional cleanup required |
| 215 | Log Suite Teardown cleanup |
| 216 | Delete All Sessions |
| 217 | |
| 218 | Setup Test |
| 219 | [Documentation] Re-create Subscriber, whitelist, and olt-device models to test |
| 220 | Log Re-creating objects |
| 221 | ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass} |
| 222 | Create Whitelist |
| 223 | Create Subscriber |
| 224 | Create VOLT |
| 225 | |
| 226 | Teardown Test |
| 227 | [Documentation] Delete xos objects, kills processes and cleans up interfaces on src+dst servers |
| 228 | Clean Up Linux |
| 229 | Clean Up XOS |
| 230 | Log Kubernetes Containers Logs Since Time ${datetime} ${container_list} |
| 231 | |
| 232 | Clean Up Linux |
| 233 | [Documentation] Kill processes and clean up interfaces on src+dst servers |
| 234 | Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 235 | Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 236 | Run Keyword If '${dst_ip}' != '${None}' Run Keyword And Ignore Error Kill Linux Process [d]hcpd ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 237 | Delete IP Addresses from Interface on Remote Host ${src_iface} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} |
| 238 | Run Keyword If '${dst_ip}' != '${None}' Delete Interface on Remote Host ${dst_dp_iface}.${s_tag} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name} |
| 239 | |
| 240 | Clean Up XOS |
| 241 | [Documentation] Clean up all XOS objects and reinstall voltha after OLT reboots |
| 242 | Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_SUBSCRIBER} |
| 243 | Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_WHITELIST} |
| 244 | Sleep 20s |
| 245 | Wait Until Keyword Succeeds 30s 2s Validate Subscriber Count 0 |
| 246 | Sleep 10s |
| 247 | Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_DEVICE} |
| 248 | Sleep 20s |
| 249 | Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_SERVICEINSTANCES} |
| 250 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip} |
| 251 | Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip} |
| 252 | Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass} |
| 253 | Sleep 15s |
| 254 | |
| 255 | Create Whitelist |
| 256 | ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0 |
| 257 | CORD Post ${ATT_WHITELIST} ${AttWhiteListDict} |
| 258 | |
| 259 | Remove Whitelist |
| 260 | ${whitelist_id}= Retrieve Whitelist Entry ${onu_device} |
| 261 | CORD Delete ${ATT_WHITELIST} ${whitelist_id} |
| 262 | |
| 263 | Update Whitelist with Wrong Location |
| 264 | ${whitelist_id}= Retrieve Whitelist Entry ${onu_device} |
| 265 | CORD Put ${ATT_WHITELIST} {"pon_port_id": 55 } ${whitelist_id} |
| 266 | |
| 267 | Update Whitelist with Correct Location |
| 268 | ${whitelist_id}= Retrieve Whitelist Entry ${onu_device} |
| 269 | CORD Put ${ATT_WHITELIST} {"pon_port_id": ${onu_location} } ${whitelist_id} |
| 270 | |
| 271 | Create Subscriber |
| 272 | ${SubscriberDict}= utils.listToDict ${SubscriberList} 0 |
| 273 | Wait Until Keyword Succeeds 120s 15s CORD Post ${VOLT_SUBSCRIBER} ${SubscriberDict} |
| 274 | |
| 275 | Remove Subscriber |
| 276 | ${subscriber_id}= Retrieve Subscriber ${c_tag} |
| 277 | CORD Delete ${VOLT_SUBSCRIBER} ${subscriber_id} |
| 278 | |
| 279 | Create VOLT |
| 280 | ${VoltDeviceDict}= utils.listToDict ${VoltDeviceList} 0 |
| 281 | CORD Post ${VOLT_DEVICE} ${VoltDeviceDict} |