blob: 5218bb85ab88743c980153def29f1c34f8ae6d1d [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053052${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000053${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
58${suppressaddsubscriber} True
59
60*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000061Verify ONU after Rebooting Physically for TT
62 [Documentation] Test the ONU functionality by physically turning on/off ONU.
63 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
64 ... Test case runs only on the PODs that are configured with PowerSwitch that
65 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
66 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
67 [Setup] Start Logging RebootOnu_PowerSwitch_TT
68 [Teardown] Run Keywords Collect Logs
69 ... AND Stop Logging RebootOnu_PowerSwitch_TT
70 ... AND Delete All Devices and Verify
71 # Add OLT device
72 Setup
73 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
74 Run Keyword If ${has_dataplane} Clean Up Linux
75 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
76 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
77 Run Keyword If ${has_dataplane} Clean Up Linux
78 FOR ${I} IN RANGE 0 ${num_all_onus}
79 ${src}= Set Variable ${hosts.src[${I}]}
80 ${dst}= Set Variable ${hosts.dst[${I}]}
81 ${service_type}= Get Variable Value ${src['service_type']} "null"
82 ${of_id}= Get ofID From OLT List ${src['olt']}
83 ${onu_device_id}= Get Device ID From SN ${src['onu']}
84 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
85 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
86 # Disable Power Switch
87 Disable Switch Outlet ${src['power_switch_port']}
88 # TODO: Add verification for MCAST
89 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
90 ... Wait Until Keyword Succeeds 60s 2s
91 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
92 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
93 # Remove Subscriber Access (To replicate TT workflow)
94 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
95 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
96 Sleep 5s
97 # Enable Power Switch
98 Enable Switch Outlet ${src['power_switch_port']}
99 # Check ONU port is Enabled in ONOS
100 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
101 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
102 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
103 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
104 ... volt-add-subscriber-access ${of_id} ${onu_port}
105 # Verify ONU state in voltha
106 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
107 ... ENABLED ACTIVE REACHABLE
108 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
109 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
110 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
111 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
112 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
113 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
114 ... ${dst['container_name']}
115 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
116 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
117 ... ${dst} ${suppressaddsubscriber}
118 END
119
120Verify OLT after Rebooting Physically for TT
121 [Documentation] Test the physical reboot of the OLT
122 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
123 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
124 [Tags] functionalTT PhysicalOltRebootTT
125 [Setup] Start Logging RebootOlt_Physical_TT
126 [Teardown] Run Keywords Collect Logs
127 ... AND Stop Logging RebootOlt_Physical_TT
128 ... AND Delete All Devices and Verify
129 # Add OLT device
130 Setup
131 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
132 Run Keyword If ${has_dataplane} Clean Up Linux
133 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
134 # Reboot the OLT from the OLT CLI
135 FOR ${I} IN RANGE 0 ${olt_count}
136 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
137 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
138 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
139 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
140 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
141 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
142 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
143 END
144 FOR ${I} IN RANGE 0 ${num_all_onus}
145 ${src}= Set Variable ${hosts.src[${I}]}
146 ${dst}= Set Variable ${hosts.dst[${I}]}
147 ${service_type}= Get Variable Value ${src['service_type']} "null"
148 # TODO: Add verification for MCAST
149 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
150 ... Wait Until Keyword Succeeds 60s 2s
151 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
152 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
153 END
154 # Wait for the OLT to come back up
155 FOR ${I} IN RANGE 0 ${olt_count}
156 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
157 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
158 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
159 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
160 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
161 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
162 ... Check Remote System Reachability True ${olt_ssh_ip}
163 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
164 ... Validate OLT Device ENABLED ACTIVE
165 ... REACHABLE ${olt_serial_number}
166 END
167 Run Keyword If ${has_dataplane} Clean Up Linux
168 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
169
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000170Verify ONU Soft Reboot for TT
171 [Documentation] Test the ONU Soft Reboot functionality.
172 [Tags] functionalTT OnuSoftRebootTT
173 [Setup] Start Logging SoftRebootOnu_TT
174 [Teardown] Run Keywords Collect Logs
175 ... AND Stop Logging SoftRebootOnu_TT
176 ... AND Delete All Devices and Verify
177 # Add OLT device
178 Setup
179 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
180 Run Keyword If ${has_dataplane} Clean Up Linux
181 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
182 FOR ${I} IN RANGE 0 ${num_all_onus}
183 ${src}= Set Variable ${hosts.src[${I}]}
184 ${dst}= Set Variable ${hosts.dst[${I}]}
185 ${service_type}= Get Variable Value ${src['service_type']} "null"
186 ${of_id}= Get ofID From OLT List ${src['olt']}
187 ${onu_device_id}= Get Device ID From SN ${src['onu']}
188 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
189 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
190 Reboot Device ${onu_device_id}
191 # TODO: Add verification for MCAST
192 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
193 ... Wait Until Keyword Succeeds ${timeout} 2s
194 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
195 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
196 # Remove Subscriber Access (To replicate TT workflow)
197 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
198 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
199 # Check ONU port is Enabled in ONOS
200 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
201 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
202 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
203 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
204 ... volt-add-subscriber-access ${of_id} ${onu_port}
205 # Verify ONU state in voltha
206 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
207 ... ENABLED ACTIVE REACHABLE
208 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
209 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
210 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
211 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
212 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
213 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
214 ... ${dst['container_name']}
215 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
216 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
217 ... ${dst} ${suppressaddsubscriber}
218 END
219 Run Keyword If ${has_dataplane} Clean Up Linux
220 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
221
Hardik Windlass656661a2021-04-23 03:46:31 +0000222Verify OLT Soft Reboot for TT
223 [Documentation] Test the OLT Soft Reboot functionality.
224 [Tags] functionalTT OltSoftRebootTT
225 [Setup] Start Logging SoftRebootOlt_TT
226 [Teardown] Run Keywords Collect Logs
227 ... AND Stop Logging SoftRebootOlt_TT
228 ... AND Delete All Devices and Verify
229 # Add OLT device
230 Setup
231 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
232 Run Keyword If ${has_dataplane} Clean Up Linux
233 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
234 # Reboot the OLT from the OLT CLI
235 FOR ${I} IN RANGE 0 ${olt_count}
236 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
237 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
238 # Reboot the OLT using "voltctl device reboot" command
239 Reboot Device ${olt_device_id}
240 END
241 FOR ${I} IN RANGE 0 ${num_all_onus}
242 ${src}= Set Variable ${hosts.src[${I}]}
243 ${dst}= Set Variable ${hosts.dst[${I}]}
244 ${service_type}= Get Variable Value ${src['service_type']} "null"
245 # TODO: Add verification for MCAST
246 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
247 ... Wait Until Keyword Succeeds ${timeout} 2s
248 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
249 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
250 END
251 # Wait for the OLT to come back up
252 FOR ${I} IN RANGE 0 ${olt_count}
253 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
254 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
255 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
256 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
257 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
258 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
259 ... Check Remote System Reachability True ${olt_ssh_ip}
260 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
261 ... Validate OLT Device ENABLED ACTIVE
262 ... REACHABLE ${olt_serial_number}
263 END
264 Run Keyword If ${has_dataplane} Clean Up Linux
265 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
266
TorstenThieme136ebf72021-02-26 14:27:07 +0000267Verify restart openonu-adapter container after subscriber provisioning for TT
268 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
269 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass04896f62021-03-05 16:15:11 +0530270 [Tags] functionalTT Restart-OpenOnu-TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000271 [Setup] Start Logging Restart-OpenOnu-TT
272 [Teardown] Run Keywords Collect Logs
273 ... AND Stop Logging Restart-OpenOnu-TT
274 ... AND Delete All Devices and Verify
275 # Add OLT device
276 Setup
277 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
278 Run Keyword If ${has_dataplane} Clean Up Linux
279 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
280 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
281 Log ${podStatusOutput}
282 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
283 ${podName} Set Variable adapter-open-onu
284 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
285 Sleep 5s
286 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
287 ... app ${podName} Running
288 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
289 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
290 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
291 Log ${podStatusOutput}
292 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
293 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
294 Log to console Pod ${podName} restarted and sanity checks passed successfully
295
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000296Verify restart openolt-adapter container before subscriber provisioning for TT
297 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
298 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
299 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
300 [Teardown] Run Keywords Collect Logs
301 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
302 ... AND Delete All Devices and Verify
303 # Add OLT device
304 Setup
305 Run Keyword If ${has_dataplane} Clean Up Linux
306 FOR ${I} IN RANGE 0 ${olt_count}
307 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
308 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
309 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
310 ... ${olt_serial_number}
311 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
312 ... Get NNI Port in ONOS ${of_id}
313 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
314 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
315 ... ${nni_port}
316 END
317 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
318 Log ${podStatusOutput}
319 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
320 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
321 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
322 Sleep 5s
323 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
324 ... app ${podName} Running
325 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
326 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
327 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
328 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
329 Log to console Pod ${podName} restarted and sanity checks passed successfully
330
Hardik Windlassd3f63892021-03-25 08:28:02 +0000331Verify restart openolt-adapter container after subscriber provisioning for TT
332 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
333 [Tags] functionalTT Restart-OpenOlt-TT
334 [Setup] Start Logging Restart-OpenOlt-TT
335 [Teardown] Run Keywords Collect Logs
336 ... AND Stop Logging Restart-OpenOlt-TT
337 Setup
338 Run Keyword If ${has_dataplane} Clean Up Linux
339 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
340 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
341 Log ${podStatusOutput}
342 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
343 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
344 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
345 Sleep 5s
346 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
347 ... app ${podName} Running
348 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
349 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
350 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
351 Log ${podStatusOutput}
352 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
353 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
354 Log to console Pod ${podName} restarted and sanity checks passed successfully
355
356Verify restart ofagent container after subscriber is provisioned for TT
357 [Documentation] Restart ofagent container after VOLTHA is operational.
358 [Tags] functionalTT ofagentRestart-TT notready
359 [Setup] Start Logging ofagentRestart-TT
360 [Teardown] Run Keywords Collect Logs
361 ... AND Stop Logging ofagentRestart-TT
362 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
363 # set timeout value
364 ${waitforRestart} Set Variable 120s
365 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
366 Log ${podStatusOutput}
367 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
368 ${podName} Set Variable ofagent
369 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
370 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
371 ... app ${podName} Running
372 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
373 Run Keyword If ${has_dataplane} Clean Up Linux
374 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
375 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
376 Log ${podStatusOutput}
377 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
378 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
379 # Scale Down the Of-Agent Deployment
380 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
381 Sleep 30s
382 FOR ${I} IN RANGE 0 ${num_all_onus}
383 ${src}= Set Variable ${hosts.src[${I}]}
384 ${dst}= Set Variable ${hosts.dst[${I}]}
385 ${of_id}= Get ofID From OLT List ${src['olt']}
386 ${onu_device_id}= Get Device ID From SN ${src['onu']}
387 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
388 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
389 # Verify ONU state in voltha
390 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
391 ... ENABLED ACTIVE REACHABLE
392 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
393 # Check ONU port is Disabled in ONOS
394 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
395 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
396 # Verify Ping
397 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
398 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
399 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
400 END
401 # Scale Up the Of-Agent Deployment
402 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
403 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
404 ... Running
405 Run Keyword If ${has_dataplane} Clean Up Linux
406 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
407 Log to console Pod ${podName} restarted and sanity checks passed successfully
408
409Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
410 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
411 ... simulate a POD crash. The test then scales the rw-core back to a single instance
412 ... and configures ONOS for access. The test succeeds if the device is able to
413 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000414 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000415 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
416 ... AND Clear All Devices Then Create New Device
417 [Teardown] Run Keywords Collect Logs
418 ... AND Stop Logging RwCoreFailAndRestart-TT
419 Run Keyword If ${has_dataplane} Clean Up Linux
420 FOR ${I} IN RANGE 0 ${olt_count}
421 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
422 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
423 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
424 ... ${olt_serial_number}
425 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
426 ... Get NNI Port in ONOS ${of_id}
427 END
428 FOR ${I} IN RANGE 0 ${num_all_onus}
429 ${src}= Set Variable ${hosts.src[${I}]}
430 ${dst}= Set Variable ${hosts.dst[${I}]}
431 ${of_id}= Get ofID From OLT List ${src['olt']}
432 ${onu_device_id}= Get Device ID From SN ${src['onu']}
433 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
434 ... ${of_id}
435 # Bring up the device and verify it authenticates
436 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
437 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded
438 END
439
440 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
441 Scale K8s Deployment voltha voltha-voltha-rw-core 0
442 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
443 # Ensure the ofagent POD goes "not-ready" as expected
444 Wait Until keyword Succeeds ${timeout} 2s
445 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
446 # Scale up the core deployment and make sure both it and the ofagent deployment are back
447 Scale K8s Deployment voltha voltha-voltha-rw-core 1
448 Wait Until Keyword Succeeds ${timeout} 2s
449 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
450 Wait Until Keyword Succeeds ${timeout} 2s
451 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
452 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
453 # so restart the port forwarding for the API service
454 Restart VOLTHA Port Forward voltha-api
455 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
456 # represents system connectivity being restored
457 FOR ${I} IN RANGE 0 ${olt_count}
458 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
459 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
460 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
461 ... ${olt_serial_number}
462 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
463 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
464 END
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000465 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000466
467*** Keywords ***
468Setup Suite
469 [Documentation] Set up the test suite
470 Common Test Suite Setup
471 ${switch_type}= Get Variable Value ${web_power_switch.type}
472 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
473
TorstenThieme136ebf72021-02-26 14:27:07 +0000474Teardown Suite
475 [Documentation] Tear down steps for the suite
476 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass04896f62021-03-05 16:15:11 +0530477 Run Keyword If ${teardown_device} Delete All Devices and Verify
Hardik Windlassd3f63892021-03-25 08:28:02 +0000478
479Clear All Devices Then Create New Device
480 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
481 # Remove all devices from voltha and onos
482 Delete All Devices and Verify
483 # Execute normal test Setup Keyword
484 Setup