Gitiles
Code Review Sign In
gerrit.opencord.org / cord-tester / a63b6deb68f47f5510a74d4d09f165c2b526df31 / . / src / test / cord-api / Tests / WorkflowValidations / DataPlane_FailureTests.robot
blob: 9b55a7bf9067b6b090de9a11d078dfd120824879 [file] [log] [blame]
Suchitra Vemurif8ef2842018-12-18 15:11:39 -0800[diff] [blame]1# Copyright 2017-present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
16Documentation Test various data plane failure scenarios with ATT workflow
17Suite Setup Setup Suite
18Suite Teardown Teardown Suite
19Test Setup Setup Test
20Test Teardown Teardown Test
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../Framework/utils/utils.py
27Resource ../../Framework/utils/utils.robot
28Library ../../Framework/restApi.py
29Resource ../../Framework/Subscriber.robot
30Resource ../../Framework/ATTWorkFlowDriver.robot
31Resource ../../Framework/Kubernetes.robot
32Resource ../../Framework/ONU.robot
33Resource ../../Framework/OLT.robot
34Resource ../../Framework/DHCP.robot
35Variables ../../Properties/RestApiProperties.py
36
37*** Variables ***
38${POD_NAME} flex-pod1-olt
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40${HELM_CHARTS_DIR} ~/helm-charts
41${WHITELIST_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTWhiteList.json
42${SUBSCRIBER_PATHFILE} ${CURDIR}/data/${POD_NAME}/ATTSubscriber.json
43${VOLT_DEVICE_PATHFILE} ${CURDIR}/data/${POD_NAME}/RealOLTDevice.json
44${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
45${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
46${VOLTHA_POD_NUM} 8
47
48*** Test Cases ***
49
50OLT Reboot
51 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
52 ... Configure whitelist with correct ONU location
53 ... Validate successful authentication/DHCP/E2E ping
54 ... Reboots OLT
55 ... Validate that pings fail
56 ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
57 [Tags] olt1
58 Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
59 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
60 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
61 Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
62 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
63 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
64 Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
65 # Reboot OLT
66 Login And Run Command On Remote System sudo reboot ${olt_ip} ${olt_user} ${olt_pass} prompt=#
67 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip}
68 Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
69 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip}
70 Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass}
71 # Validate successful pings since the OLT is Up
72 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
73 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
74 Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
75
Suchitra Vemuri197fbf82019-01-03 14:06:42 -0800[diff] [blame]76Fabric Switch Reboot
77 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
78 ... Configure whitelist with correct ONU location
79 ... Validate successful authentication/DHCP/E2E ping
80 ... Reboots Fabric Switch
81 ... Validate that pings fail
82 ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
83 #[Setup] None
Suchitra Vemuri64d5c862019-01-07 19:09:11 -0800[diff] [blame]84 #[Teardown] None
Suchitra Vemuri197fbf82019-01-03 14:06:42 -0800[diff] [blame]85 [Tags] fabric1
86 Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
87 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
88 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
89 Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
90 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
91 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
92 Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
93 # Reboot Fabric Switch
94 Login And Run Command On Remote System sudo reboot ${fabric_ip} ${fabric_user} ${fabric_pass} prompt=#
95 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip}
96 Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
97 #Validate DHCP and Ping False False ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
98 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${fabric_ip}
99 # Validate successful pings since Fabric Switch is Up
100 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
101 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
102 Wait Until Keyword Succeeds 60s 2s Check Ping True ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
Suchitra Vemurif8ef2842018-12-18 15:11:39 -0800[diff] [blame]103
Suchitra Vemuri64d5c862019-01-07 19:09:11 -0800[diff] [blame]104Subscriber(RG) Reboot
105 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
106 ... Configure whitelist with correct ONU location
107 ... Validate successful authentication/DHCP/E2E ping
108 ... Reboots RG
109 ... Validate that pings fail
110 ... Validate successful authentication/DHCP/E2E ping after OLT comes back up
111 [Tags] RG-reboot
112 Wait Until Keyword Succeeds 300s 15s Validate ONU States ACTIVE ENABLED ${onu_device}
113 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
114 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
115 Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
116 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
117 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
118 Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
119 # Reboot RG
120 Login And Run Command On Remote System sudo reboot ${src_ip} ${src_user} ${src_pass} prompt=$
121 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${fabric_ip}
122 Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
123 Wait Until Keyword Succeeds 150s 10s Check Remote System Reachability True ${src_ip}
124 Wait Until Keyword Succeeds 60s 2s Check Ping False ${dst_dp_ip} ${src_iface} ${src_ip} ${src_user} ${src_pass}
125 # Perform Reauthentication/DHCP and Ping
126 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED AWAITING ${onu_device}
127 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status awaiting-auth ${onu_device}
128 Validate Authentication True ${src_iface} wpa_supplicant.conf ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
129 Wait Until Keyword Succeeds 60s 2s Validate ATT Workflow Driver SI ENABLED APPROVED ${onu_device}
130 Wait Until Keyword Succeeds 60s 2s Validate Subscriber Status enabled ${onu_device}
131 Validate DHCP and Ping True True ${src_iface} ${s_tag} ${c_tag} ${dst_dp_ip} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name} ${dst_dp_iface} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
132
Suchitra Vemurif8ef2842018-12-18 15:11:39 -0800[diff] [blame]133*** Keywords ***
134Setup Suite
135 ${auth} = Create List ${XOS_USER} ${XOS_PASSWD}
136 ${HEADERS} Create Dictionary Content-Type=application/json
137 Create Session ${server_ip} http://${server_ip}:${server_port} auth=${AUTH} headers=${HEADERS}
138 ${att_workflow_service_id}= Get Service Owner Id ${ATT_SERVICE}
139 ${volt_service_id}= Get Service Owner Id ${VOLT_SERVICE}
140 ${AttWhiteListList}= utils.jsonToList ${WHITELIST_PATHFILE} AttWhiteListInfo
141 Set Suite Variable ${AttWhiteListList}
142 ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0
143 ${AttWhiteListDict}= utils.setFieldValueInDict ${AttWhiteListDict} owner_id ${att_workflow_service_id}
144 ${onu_device}= Get From Dictionary ${AttWhiteListDict} serial_number
145 Set Global Variable ${onu_device}
146 ${onu_location}= Get From Dictionary ${AttWhiteListDict} pon_port_id
147 Set Global Variable ${onu_location}
148 ${SubscriberList}= utils.jsonToList ${SUBSCRIBER_PATHFILE} SubscriberInfo
149 Set Global Variable ${SubscriberList}
150 ${SubscriberDict}= utils.listToDict ${SubscriberList} 0
151 ${s_tag}= utils.getFieldValueFromDict ${SubscriberDict} s_tag
152 ${c_tag}= utils.getFieldValueFromDict ${SubscriberDict} c_tag
153 ${VoltDeviceList}= utils.jsonToList ${VOLT_DEVICE_PATHFILE} VOLTDeviceInfo
154 Set Global Variable ${VoltDeviceList}
155 Set Suite Variable ${s_tag}
156 Set Suite Variable ${c_tag}
157 Set Global Variable ${export_kubeconfig} export KUBECONFIG=${KUBERNETES_CONF}
158 # Read variables from yaml file
159 ${src_ip}= Evaluate ${hosts}.get("src").get("ip")
160 ${src_user}= Evaluate ${hosts}.get("src").get("user")
161 ${src_pass}= Evaluate ${hosts}.get("src").get("pass")
162 ${src_container_type}= Evaluate ${hosts}.get("src").get("container_type")
163 ${src_container_name}= Evaluate ${hosts}.get("src").get("container_name")
164 ${src_iface}= Evaluate ${hosts}.get("src").get("dp_iface_name")
165 ${dst_ip}= Evaluate ${hosts}.get("dst").get("ip")
166 ${dst_user} = Evaluate ${hosts}.get("dst").get("user")
167 ${dst_pass}= Evaluate ${hosts}.get("dst").get("pass")
168 ${dst_container_type}= Evaluate ${hosts}.get("dst").get("container_type")
169 ${dst_container_name}= Evaluate ${hosts}.get("dst").get("container_name")
170 ${dst_dp_iface}= Evaluate ${hosts}.get("dst").get("dp_iface_name")
171 ${dst_dp_ip}= Evaluate ${hosts}.get("dst").get("dp_iface_ip_qinq")
172 ${olt_ip}= Evaluate ${olts}[0].get("ip")
173 ${olt_user}= Evaluate ${olts}[0].get("user")
174 ${olt_pass}= Evaluate ${olts}[0].get("pass")
175 ${fabric_ip}= Evaluate ${fabric_switches}[0].get("ip")
176 ${fabric_user}= Evaluate ${fabric_switches}[0].get("user")
177 ${fabric_pass}= Evaluate ${fabric_switches}[0].get("pass")
178 ${k8s_node_ip}= Evaluate ${nodes}[0].get("ip")
179 ${k8s_node_user}= Evaluate ${nodes}[0].get("user")
180 ${k8s_node_pass}= Evaluate ${nodes}[0].get("pass")
181 Set Suite Variable ${src_ip}
182 Set Suite Variable ${src_user}
183 Set Suite Variable ${src_pass}
184 Set Suite Variable ${src_container_type}
185 Set Suite Variable ${src_container_name}
186 Set Suite Variable ${src_iface}
187 Set Suite Variable ${dst_ip}
188 Set Suite Variable ${dst_user}
189 Set Suite Variable ${dst_pass}
190 Set Suite Variable ${dst_container_type}
191 Set Suite Variable ${dst_container_name}
192 Set Suite Variable ${dst_dp_iface}
193 Set Suite Variable ${dst_dp_ip}
194 Set Suite Variable ${olt_ip}
195 Set Suite Variable ${olt_user}
196 Set Suite Variable ${olt_pass}
197 Set Suite Variable ${fabric_ip}
198 Set Suite Variable ${fabric_user}
199 Set Suite Variable ${fabric_pass}
200 Set Suite Variable ${k8s_node_ip}
201 Set Suite Variable ${k8s_node_user}
202 Set Suite Variable ${k8s_node_pass}
203 @{container_list}= Create List
204 Append To List ${container_list} att-workflow-att-workflow-driver
205 Append To List ${container_list} att-workflow-volt
206 Append To List ${container_list} onos
207 Append To List ${container_list} xos-core
208 Append To List ${container_list} vcore
209 Set Suite Variable ${container_list}
210 ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass}
211 Set Suite Variable ${datetime}
212
213Teardown Suite
214 [Documentation] Performs any additional cleanup required
215 Log Suite Teardown cleanup
216 Delete All Sessions
217
218Setup Test
219 [Documentation] Re-create Subscriber, whitelist, and olt-device models to test
220 Log Re-creating objects
221 ${datetime}= Get Current Datetime On Kubernetes Node ${k8s_node_ip} ${k8s_node_user} ${k8s_node_pass}
222 Create Whitelist
223 Create Subscriber
224 Create VOLT
225
226Teardown Test
227 [Documentation] Delete xos objects, kills processes and cleans up interfaces on src+dst servers
228 Clean Up Linux
229 Clean Up XOS
230 Log Kubernetes Containers Logs Since Time ${datetime} ${container_list}
231
232Clean Up Linux
233 [Documentation] Kill processes and clean up interfaces on src+dst servers
234 Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
235 Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
236 Run Keyword If '${dst_ip}' != '${None}' Run Keyword And Ignore Error Kill Linux Process [d]hcpd ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
237 Delete IP Addresses from Interface on Remote Host ${src_iface} ${src_ip} ${src_user} ${src_pass} ${src_container_type} ${src_container_name}
238 Run Keyword If '${dst_ip}' != '${None}' Delete Interface on Remote Host ${dst_dp_iface}.${s_tag} ${dst_ip} ${dst_user} ${dst_pass} ${dst_container_type} ${dst_container_name}
239
240Clean Up XOS
241 [Documentation] Clean up all XOS objects and reinstall voltha after OLT reboots
242 Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_SUBSCRIBER}
243 Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_WHITELIST}
244 Sleep 20s
245 Wait Until Keyword Succeeds 30s 2s Validate Subscriber Count 0
246 Sleep 10s
247 Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${VOLT_DEVICE}
248 Sleep 20s
249 Wait Until Keyword Succeeds 60s 2s Clean Up Objects ${ATT_SERVICEINSTANCES}
250 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability False ${olt_ip}
251 Wait Until Keyword Succeeds 120s 10s Check Remote System Reachability True ${olt_ip}
252 Wait Until Keyword Succeeds 120s 10s Openolt is Up ${olt_ip} ${olt_user} ${olt_pass}
253 Sleep 15s
254
255Create Whitelist
256 ${AttWhiteListDict}= utils.listToDict ${AttWhiteListList} 0
257 CORD Post ${ATT_WHITELIST} ${AttWhiteListDict}
258
259Remove Whitelist
260 ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
261 CORD Delete ${ATT_WHITELIST} ${whitelist_id}
262
263Update Whitelist with Wrong Location
264 ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
265 CORD Put ${ATT_WHITELIST} {"pon_port_id": 55 } ${whitelist_id}
266
267Update Whitelist with Correct Location
268 ${whitelist_id}= Retrieve Whitelist Entry ${onu_device}
269 CORD Put ${ATT_WHITELIST} {"pon_port_id": ${onu_location} } ${whitelist_id}
270
271Create Subscriber
272 ${SubscriberDict}= utils.listToDict ${SubscriberList} 0
273 Wait Until Keyword Succeeds 120s 15s CORD Post ${VOLT_SUBSCRIBER} ${SubscriberDict}
274
275Remove Subscriber
276 ${subscriber_id}= Retrieve Subscriber ${c_tag}
277 CORD Delete ${VOLT_SUBSCRIBER} ${subscriber_id}
278
279Create VOLT
280 ${VoltDeviceDict}= utils.listToDict ${VoltDeviceList} 0
281 CORD Post ${VOLT_DEVICE} ${VoltDeviceDict}