blob: c4a656e33cb4a4cf357ad05e82e52d8f2e724cca [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053052${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000053${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070058# logging flag to enable Collect Logs, can be passed via the command line too
59# example: -v logging:False
60${logging} True
61
TorstenThieme136ebf72021-02-26 14:27:07 +000062${suppressaddsubscriber} True
63
64*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000065Verify ONU after Rebooting Physically for TT
66 [Documentation] Test the ONU functionality by physically turning on/off ONU.
67 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
68 ... Test case runs only on the PODs that are configured with PowerSwitch that
69 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
70 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
71 [Setup] Start Logging RebootOnu_PowerSwitch_TT
72 [Teardown] Run Keywords Collect Logs
73 ... AND Stop Logging RebootOnu_PowerSwitch_TT
74 ... AND Delete All Devices and Verify
75 # Add OLT device
76 Setup
77 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
78 Run Keyword If ${has_dataplane} Clean Up Linux
79 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
80 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
81 Run Keyword If ${has_dataplane} Clean Up Linux
82 FOR ${I} IN RANGE 0 ${num_all_onus}
83 ${src}= Set Variable ${hosts.src[${I}]}
84 ${dst}= Set Variable ${hosts.dst[${I}]}
85 ${service_type}= Get Variable Value ${src['service_type']} "null"
86 ${of_id}= Get ofID From OLT List ${src['olt']}
87 ${onu_device_id}= Get Device ID From SN ${src['onu']}
88 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
89 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
90 # Disable Power Switch
91 Disable Switch Outlet ${src['power_switch_port']}
92 # TODO: Add verification for MCAST
93 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
94 ... Wait Until Keyword Succeeds 60s 2s
95 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
96 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
97 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +000098 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass364ec452021-04-14 07:12:28 +000099 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
100 Sleep 5s
101 # Enable Power Switch
102 Enable Switch Outlet ${src['power_switch_port']}
103 # Check ONU port is Enabled in ONOS
104 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000105 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass364ec452021-04-14 07:12:28 +0000106 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000107 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass364ec452021-04-14 07:12:28 +0000108 ... volt-add-subscriber-access ${of_id} ${onu_port}
109 # Verify ONU state in voltha
110 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
111 ... ENABLED ACTIVE REACHABLE
112 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
113 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
114 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
115 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
116 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
117 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
118 ... ${dst['container_name']}
119 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
120 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
121 ... ${dst} ${suppressaddsubscriber}
122 END
123
124Verify OLT after Rebooting Physically for TT
125 [Documentation] Test the physical reboot of the OLT
126 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
127 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
128 [Tags] functionalTT PhysicalOltRebootTT
129 [Setup] Start Logging RebootOlt_Physical_TT
130 [Teardown] Run Keywords Collect Logs
131 ... AND Stop Logging RebootOlt_Physical_TT
132 ... AND Delete All Devices and Verify
133 # Add OLT device
134 Setup
135 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
136 Run Keyword If ${has_dataplane} Clean Up Linux
137 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
138 # Reboot the OLT from the OLT CLI
139 FOR ${I} IN RANGE 0 ${olt_count}
140 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
141 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
142 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
143 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
144 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
145 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
146 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
147 END
148 FOR ${I} IN RANGE 0 ${num_all_onus}
149 ${src}= Set Variable ${hosts.src[${I}]}
150 ${dst}= Set Variable ${hosts.dst[${I}]}
151 ${service_type}= Get Variable Value ${src['service_type']} "null"
152 # TODO: Add verification for MCAST
153 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
154 ... Wait Until Keyword Succeeds 60s 2s
155 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
156 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
157 END
158 # Wait for the OLT to come back up
159 FOR ${I} IN RANGE 0 ${olt_count}
160 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
161 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
162 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
163 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
164 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
165 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
166 ... Check Remote System Reachability True ${olt_ssh_ip}
167 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
168 ... Validate OLT Device ENABLED ACTIVE
169 ... REACHABLE ${olt_serial_number}
170 END
171 Run Keyword If ${has_dataplane} Clean Up Linux
172 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
173
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000174Verify ONU Soft Reboot for TT
175 [Documentation] Test the ONU Soft Reboot functionality.
176 [Tags] functionalTT OnuSoftRebootTT
177 [Setup] Start Logging SoftRebootOnu_TT
178 [Teardown] Run Keywords Collect Logs
179 ... AND Stop Logging SoftRebootOnu_TT
180 ... AND Delete All Devices and Verify
181 # Add OLT device
182 Setup
183 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
184 Run Keyword If ${has_dataplane} Clean Up Linux
185 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
186 FOR ${I} IN RANGE 0 ${num_all_onus}
187 ${src}= Set Variable ${hosts.src[${I}]}
188 ${dst}= Set Variable ${hosts.dst[${I}]}
189 ${service_type}= Get Variable Value ${src['service_type']} "null"
190 ${of_id}= Get ofID From OLT List ${src['olt']}
191 ${onu_device_id}= Get Device ID From SN ${src['onu']}
192 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
193 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
194 Reboot Device ${onu_device_id}
195 # TODO: Add verification for MCAST
196 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
197 ... Wait Until Keyword Succeeds ${timeout} 2s
198 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
199 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
200 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000201 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000202 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
203 # Check ONU port is Enabled in ONOS
204 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000205 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000206 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000207 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000208 ... volt-add-subscriber-access ${of_id} ${onu_port}
209 # Verify ONU state in voltha
210 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
211 ... ENABLED ACTIVE REACHABLE
212 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
213 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
214 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
215 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
216 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
217 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
218 ... ${dst['container_name']}
219 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
220 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
221 ... ${dst} ${suppressaddsubscriber}
222 END
223 Run Keyword If ${has_dataplane} Clean Up Linux
224 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
225
Hardik Windlass656661a2021-04-23 03:46:31 +0000226Verify OLT Soft Reboot for TT
227 [Documentation] Test the OLT Soft Reboot functionality.
228 [Tags] functionalTT OltSoftRebootTT
229 [Setup] Start Logging SoftRebootOlt_TT
230 [Teardown] Run Keywords Collect Logs
231 ... AND Stop Logging SoftRebootOlt_TT
232 ... AND Delete All Devices and Verify
233 # Add OLT device
234 Setup
235 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
236 Run Keyword If ${has_dataplane} Clean Up Linux
237 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
238 # Reboot the OLT from the OLT CLI
239 FOR ${I} IN RANGE 0 ${olt_count}
240 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
241 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
242 # Reboot the OLT using "voltctl device reboot" command
243 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200244 # Wait for the OLT to actually go down
245 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
246 ... ${olt_serial_number}
Hardik Windlass656661a2021-04-23 03:46:31 +0000247 END
248 FOR ${I} IN RANGE 0 ${num_all_onus}
249 ${src}= Set Variable ${hosts.src[${I}]}
250 ${dst}= Set Variable ${hosts.dst[${I}]}
251 ${service_type}= Get Variable Value ${src['service_type']} "null"
252 # TODO: Add verification for MCAST
253 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
254 ... Wait Until Keyword Succeeds ${timeout} 2s
255 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
256 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
257 END
258 # Wait for the OLT to come back up
259 FOR ${I} IN RANGE 0 ${olt_count}
260 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
261 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
262 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
263 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
264 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
265 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
266 ... Check Remote System Reachability True ${olt_ssh_ip}
267 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
268 ... Validate OLT Device ENABLED ACTIVE
269 ... REACHABLE ${olt_serial_number}
270 END
271 Run Keyword If ${has_dataplane} Clean Up Linux
272 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
273
TorstenThieme136ebf72021-02-26 14:27:07 +0000274Verify restart openonu-adapter container after subscriber provisioning for TT
275 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
276 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass04896f62021-03-05 16:15:11 +0530277 [Tags] functionalTT Restart-OpenOnu-TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000278 [Setup] Start Logging Restart-OpenOnu-TT
279 [Teardown] Run Keywords Collect Logs
280 ... AND Stop Logging Restart-OpenOnu-TT
281 ... AND Delete All Devices and Verify
282 # Add OLT device
283 Setup
284 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
285 Run Keyword If ${has_dataplane} Clean Up Linux
286 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
287 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
288 Log ${podStatusOutput}
289 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
290 ${podName} Set Variable adapter-open-onu
291 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
292 Sleep 5s
293 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
294 ... app ${podName} Running
295 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
296 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
297 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
298 Log ${podStatusOutput}
299 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
300 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
301 Log to console Pod ${podName} restarted and sanity checks passed successfully
302
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000303Verify restart openolt-adapter container before subscriber provisioning for TT
304 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
305 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
306 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
307 [Teardown] Run Keywords Collect Logs
308 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
309 ... AND Delete All Devices and Verify
310 # Add OLT device
311 Setup
312 Run Keyword If ${has_dataplane} Clean Up Linux
313 FOR ${I} IN RANGE 0 ${olt_count}
314 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
315 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
316 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
317 ... ${olt_serial_number}
318 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
319 ... Get NNI Port in ONOS ${of_id}
320 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
321 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
322 ... ${nni_port}
323 END
324 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
325 Log ${podStatusOutput}
326 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
327 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
328 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
329 Sleep 5s
330 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
331 ... app ${podName} Running
332 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
333 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
334 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
335 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
336 Log to console Pod ${podName} restarted and sanity checks passed successfully
337
Hardik Windlassd3f63892021-03-25 08:28:02 +0000338Verify restart openolt-adapter container after subscriber provisioning for TT
339 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
340 [Tags] functionalTT Restart-OpenOlt-TT
341 [Setup] Start Logging Restart-OpenOlt-TT
342 [Teardown] Run Keywords Collect Logs
343 ... AND Stop Logging Restart-OpenOlt-TT
344 Setup
345 Run Keyword If ${has_dataplane} Clean Up Linux
346 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
347 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
348 Log ${podStatusOutput}
349 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
350 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
351 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
352 Sleep 5s
353 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
354 ... app ${podName} Running
355 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
356 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
357 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
358 Log ${podStatusOutput}
359 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
360 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
361 Log to console Pod ${podName} restarted and sanity checks passed successfully
362
363Verify restart ofagent container after subscriber is provisioned for TT
364 [Documentation] Restart ofagent container after VOLTHA is operational.
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000365 [Tags] functionalTT ofagentRestart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000366 [Setup] Start Logging ofagentRestart-TT
367 [Teardown] Run Keywords Collect Logs
368 ... AND Stop Logging ofagentRestart-TT
369 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
370 # set timeout value
371 ${waitforRestart} Set Variable 120s
372 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
373 Log ${podStatusOutput}
374 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
375 ${podName} Set Variable ofagent
376 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
377 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
378 ... app ${podName} Running
379 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
380 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000381 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000382 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
383 Log ${podStatusOutput}
384 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
385 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
386 # Scale Down the Of-Agent Deployment
387 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
388 Sleep 30s
389 FOR ${I} IN RANGE 0 ${num_all_onus}
390 ${src}= Set Variable ${hosts.src[${I}]}
391 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000392 ${service_type}= Get Variable Value ${src['service_type']} "null"
Hardik Windlassd3f63892021-03-25 08:28:02 +0000393 ${of_id}= Get ofID From OLT List ${src['olt']}
394 ${onu_device_id}= Get Device ID From SN ${src['onu']}
395 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
396 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
397 # Verify ONU state in voltha
398 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
399 ... ENABLED ACTIVE REACHABLE
400 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
401 # Check ONU port is Disabled in ONOS
402 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000403 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000404 # Verify Ping
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000405 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
406 ... Run Keyword And Continue On Failure Check Ping True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000407 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
408 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
409 END
410 # Scale Up the Of-Agent Deployment
411 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
412 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
413 ... Running
414 Run Keyword If ${has_dataplane} Clean Up Linux
415 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
416 Log to console Pod ${podName} restarted and sanity checks passed successfully
417
418Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
419 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
420 ... simulate a POD crash. The test then scales the rw-core back to a single instance
421 ... and configures ONOS for access. The test succeeds if the device is able to
422 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000423 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000424 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
425 ... AND Clear All Devices Then Create New Device
426 [Teardown] Run Keywords Collect Logs
427 ... AND Stop Logging RwCoreFailAndRestart-TT
428 Run Keyword If ${has_dataplane} Clean Up Linux
429 FOR ${I} IN RANGE 0 ${olt_count}
430 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
431 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
432 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
433 ... ${olt_serial_number}
434 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
435 ... Get NNI Port in ONOS ${of_id}
436 END
437 FOR ${I} IN RANGE 0 ${num_all_onus}
438 ${src}= Set Variable ${hosts.src[${I}]}
439 ${dst}= Set Variable ${hosts.dst[${I}]}
440 ${of_id}= Get ofID From OLT List ${src['olt']}
441 ${onu_device_id}= Get Device ID From SN ${src['onu']}
442 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
443 ... ${of_id}
444 # Bring up the device and verify it authenticates
445 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000446 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000447 END
448
449 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
450 Scale K8s Deployment voltha voltha-voltha-rw-core 0
451 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
452 # Ensure the ofagent POD goes "not-ready" as expected
453 Wait Until keyword Succeeds ${timeout} 2s
454 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
455 # Scale up the core deployment and make sure both it and the ofagent deployment are back
456 Scale K8s Deployment voltha voltha-voltha-rw-core 1
457 Wait Until Keyword Succeeds ${timeout} 2s
458 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
459 Wait Until Keyword Succeeds ${timeout} 2s
460 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
461 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
462 # so restart the port forwarding for the API service
463 Restart VOLTHA Port Forward voltha-api
464 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
465 # represents system connectivity being restored
466 FOR ${I} IN RANGE 0 ${olt_count}
467 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
468 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
469 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
470 ... ${olt_serial_number}
471 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
472 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
473 END
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000474 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000475
476*** Keywords ***
477Setup Suite
478 [Documentation] Set up the test suite
479 Common Test Suite Setup
480 ${switch_type}= Get Variable Value ${web_power_switch.type}
481 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
482
TorstenThieme136ebf72021-02-26 14:27:07 +0000483Teardown Suite
484 [Documentation] Tear down steps for the suite
485 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass04896f62021-03-05 16:15:11 +0530486 Run Keyword If ${teardown_device} Delete All Devices and Verify
TorstenThieme4e2168e2021-06-22 14:01:47 +0000487 Close All ONOS SSH Connections
Hardik Windlassd3f63892021-03-25 08:28:02 +0000488
489Clear All Devices Then Create New Device
490 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
491 # Remove all devices from voltha and onos
492 Delete All Devices and Verify
493 # Execute normal test Setup Keyword
494 Setup