blob: 0ec29098a5e85a8dae64f419ef1ceaa7ee917a5d [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Matteo Scandoloa4e59522021-08-31 14:56:24 -070045${STACK_NAME} voltha
TorstenThieme136ebf72021-02-26 14:27:07 +000046# For below variable value, using deployment name as using grep for
47# parsing radius pod name, we can also use full radius pod name
48${RESTART_POD_NAME} radius
49${timeout} 60s
50${of_id} 0
51${logical_id} 0
52${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053053${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000054${scripts} ../../scripts
55
56# Per-test logging on failure is turned off by default; set this variable to enable
57${container_log_dir} ${None}
58
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070059# logging flag to enable Collect Logs, can be passed via the command line too
60# example: -v logging:False
61${logging} True
62
TorstenThieme136ebf72021-02-26 14:27:07 +000063${suppressaddsubscriber} True
64
65*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000066Verify ONU after Rebooting Physically for TT
67 [Documentation] Test the ONU functionality by physically turning on/off ONU.
68 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
69 ... Test case runs only on the PODs that are configured with PowerSwitch that
70 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
71 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
72 [Setup] Start Logging RebootOnu_PowerSwitch_TT
73 [Teardown] Run Keywords Collect Logs
74 ... AND Stop Logging RebootOnu_PowerSwitch_TT
75 ... AND Delete All Devices and Verify
76 # Add OLT device
77 Setup
78 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
79 Run Keyword If ${has_dataplane} Clean Up Linux
80 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
81 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
82 Run Keyword If ${has_dataplane} Clean Up Linux
83 FOR ${I} IN RANGE 0 ${num_all_onus}
84 ${src}= Set Variable ${hosts.src[${I}]}
85 ${dst}= Set Variable ${hosts.dst[${I}]}
86 ${service_type}= Get Variable Value ${src['service_type']} "null"
87 ${of_id}= Get ofID From OLT List ${src['olt']}
88 ${onu_device_id}= Get Device ID From SN ${src['onu']}
89 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
90 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
91 # Disable Power Switch
92 Disable Switch Outlet ${src['power_switch_port']}
93 # TODO: Add verification for MCAST
94 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
95 ... Wait Until Keyword Succeeds 60s 2s
96 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
97 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
98 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +000099 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass364ec452021-04-14 07:12:28 +0000100 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
101 Sleep 5s
102 # Enable Power Switch
103 Enable Switch Outlet ${src['power_switch_port']}
104 # Check ONU port is Enabled in ONOS
105 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000106 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass364ec452021-04-14 07:12:28 +0000107 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000108 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass364ec452021-04-14 07:12:28 +0000109 ... volt-add-subscriber-access ${of_id} ${onu_port}
110 # Verify ONU state in voltha
111 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
112 ... ENABLED ACTIVE REACHABLE
113 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
114 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
115 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
116 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
117 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
118 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
119 ... ${dst['container_name']}
120 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
121 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
122 ... ${dst} ${suppressaddsubscriber}
123 END
124
125Verify OLT after Rebooting Physically for TT
126 [Documentation] Test the physical reboot of the OLT
127 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
128 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
129 [Tags] functionalTT PhysicalOltRebootTT
130 [Setup] Start Logging RebootOlt_Physical_TT
131 [Teardown] Run Keywords Collect Logs
132 ... AND Stop Logging RebootOlt_Physical_TT
133 ... AND Delete All Devices and Verify
134 # Add OLT device
135 Setup
136 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
137 Run Keyword If ${has_dataplane} Clean Up Linux
138 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
139 # Reboot the OLT from the OLT CLI
140 FOR ${I} IN RANGE 0 ${olt_count}
141 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
142 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
143 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
144 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
145 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
146 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
147 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
148 END
149 FOR ${I} IN RANGE 0 ${num_all_onus}
150 ${src}= Set Variable ${hosts.src[${I}]}
151 ${dst}= Set Variable ${hosts.dst[${I}]}
152 ${service_type}= Get Variable Value ${src['service_type']} "null"
153 # TODO: Add verification for MCAST
154 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
155 ... Wait Until Keyword Succeeds 60s 2s
156 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
157 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
158 END
159 # Wait for the OLT to come back up
160 FOR ${I} IN RANGE 0 ${olt_count}
161 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
162 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
163 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
164 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
165 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
166 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
167 ... Check Remote System Reachability True ${olt_ssh_ip}
168 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
169 ... Validate OLT Device ENABLED ACTIVE
170 ... REACHABLE ${olt_serial_number}
171 END
172 Run Keyword If ${has_dataplane} Clean Up Linux
173 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
174
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000175Verify ONU Soft Reboot for TT
176 [Documentation] Test the ONU Soft Reboot functionality.
177 [Tags] functionalTT OnuSoftRebootTT
178 [Setup] Start Logging SoftRebootOnu_TT
179 [Teardown] Run Keywords Collect Logs
180 ... AND Stop Logging SoftRebootOnu_TT
181 ... AND Delete All Devices and Verify
182 # Add OLT device
183 Setup
184 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
185 Run Keyword If ${has_dataplane} Clean Up Linux
186 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
187 FOR ${I} IN RANGE 0 ${num_all_onus}
188 ${src}= Set Variable ${hosts.src[${I}]}
189 ${dst}= Set Variable ${hosts.dst[${I}]}
190 ${service_type}= Get Variable Value ${src['service_type']} "null"
191 ${of_id}= Get ofID From OLT List ${src['olt']}
192 ${onu_device_id}= Get Device ID From SN ${src['onu']}
193 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
194 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
195 Reboot Device ${onu_device_id}
196 # TODO: Add verification for MCAST
197 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
198 ... Wait Until Keyword Succeeds ${timeout} 2s
199 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
200 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
201 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000202 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000203 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
204 # Check ONU port is Enabled in ONOS
205 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000206 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000207 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000208 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000209 ... volt-add-subscriber-access ${of_id} ${onu_port}
210 # Verify ONU state in voltha
211 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
212 ... ENABLED ACTIVE REACHABLE
213 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
214 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
215 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
216 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
217 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
218 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
219 ... ${dst['container_name']}
220 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
221 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
222 ... ${dst} ${suppressaddsubscriber}
223 END
224 Run Keyword If ${has_dataplane} Clean Up Linux
225 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
226
Hardik Windlass656661a2021-04-23 03:46:31 +0000227Verify OLT Soft Reboot for TT
228 [Documentation] Test the OLT Soft Reboot functionality.
229 [Tags] functionalTT OltSoftRebootTT
230 [Setup] Start Logging SoftRebootOlt_TT
231 [Teardown] Run Keywords Collect Logs
232 ... AND Stop Logging SoftRebootOlt_TT
233 ... AND Delete All Devices and Verify
234 # Add OLT device
235 Setup
236 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
237 Run Keyword If ${has_dataplane} Clean Up Linux
238 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
239 # Reboot the OLT from the OLT CLI
240 FOR ${I} IN RANGE 0 ${olt_count}
241 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
242 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
243 # Reboot the OLT using "voltctl device reboot" command
244 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200245 # Wait for the OLT to actually go down
246 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
247 ... ${olt_serial_number}
Hardik Windlass656661a2021-04-23 03:46:31 +0000248 END
249 FOR ${I} IN RANGE 0 ${num_all_onus}
250 ${src}= Set Variable ${hosts.src[${I}]}
251 ${dst}= Set Variable ${hosts.dst[${I}]}
252 ${service_type}= Get Variable Value ${src['service_type']} "null"
253 # TODO: Add verification for MCAST
254 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
255 ... Wait Until Keyword Succeeds ${timeout} 2s
256 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
257 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
258 END
259 # Wait for the OLT to come back up
260 FOR ${I} IN RANGE 0 ${olt_count}
261 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
262 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
263 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
264 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
265 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
266 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
267 ... Check Remote System Reachability True ${olt_ssh_ip}
268 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
269 ... Validate OLT Device ENABLED ACTIVE
270 ... REACHABLE ${olt_serial_number}
271 END
272 Run Keyword If ${has_dataplane} Clean Up Linux
273 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
274
TorstenThieme136ebf72021-02-26 14:27:07 +0000275Verify restart openonu-adapter container after subscriber provisioning for TT
276 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
277 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass04896f62021-03-05 16:15:11 +0530278 [Tags] functionalTT Restart-OpenOnu-TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000279 [Setup] Start Logging Restart-OpenOnu-TT
280 [Teardown] Run Keywords Collect Logs
281 ... AND Stop Logging Restart-OpenOnu-TT
282 ... AND Delete All Devices and Verify
283 # Add OLT device
284 Setup
285 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
286 Run Keyword If ${has_dataplane} Clean Up Linux
287 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
288 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
289 Log ${podStatusOutput}
290 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
291 ${podName} Set Variable adapter-open-onu
292 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
293 Sleep 5s
294 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
295 ... app ${podName} Running
296 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
297 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
298 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
299 Log ${podStatusOutput}
300 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
301 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
302 Log to console Pod ${podName} restarted and sanity checks passed successfully
303
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000304Verify restart openolt-adapter container before subscriber provisioning for TT
305 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
306 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
307 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
308 [Teardown] Run Keywords Collect Logs
309 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
310 ... AND Delete All Devices and Verify
311 # Add OLT device
312 Setup
313 Run Keyword If ${has_dataplane} Clean Up Linux
314 FOR ${I} IN RANGE 0 ${olt_count}
315 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
316 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
317 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
318 ... ${olt_serial_number}
319 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
320 ... Get NNI Port in ONOS ${of_id}
321 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
322 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
323 ... ${nni_port}
324 END
325 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
326 Log ${podStatusOutput}
327 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
328 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
329 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
330 Sleep 5s
331 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
332 ... app ${podName} Running
333 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
334 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
335 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
336 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
337 Log to console Pod ${podName} restarted and sanity checks passed successfully
338
Hardik Windlassd3f63892021-03-25 08:28:02 +0000339Verify restart openolt-adapter container after subscriber provisioning for TT
340 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
341 [Tags] functionalTT Restart-OpenOlt-TT
342 [Setup] Start Logging Restart-OpenOlt-TT
343 [Teardown] Run Keywords Collect Logs
344 ... AND Stop Logging Restart-OpenOlt-TT
345 Setup
346 Run Keyword If ${has_dataplane} Clean Up Linux
347 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
348 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
349 Log ${podStatusOutput}
350 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
351 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
352 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
353 Sleep 5s
354 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
355 ... app ${podName} Running
356 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
357 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
358 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
359 Log ${podStatusOutput}
360 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
361 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
362 Log to console Pod ${podName} restarted and sanity checks passed successfully
363
364Verify restart ofagent container after subscriber is provisioned for TT
365 [Documentation] Restart ofagent container after VOLTHA is operational.
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000366 [Tags] functionalTT ofagentRestart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000367 [Setup] Start Logging ofagentRestart-TT
368 [Teardown] Run Keywords Collect Logs
369 ... AND Stop Logging ofagentRestart-TT
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700370 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000371 # set timeout value
372 ${waitforRestart} Set Variable 120s
373 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
374 Log ${podStatusOutput}
375 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
376 ${podName} Set Variable ofagent
377 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
378 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
379 ... app ${podName} Running
380 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
381 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000382 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000383 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
384 Log ${podStatusOutput}
385 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
386 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
387 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700388 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlassd3f63892021-03-25 08:28:02 +0000389 Sleep 30s
390 FOR ${I} IN RANGE 0 ${num_all_onus}
391 ${src}= Set Variable ${hosts.src[${I}]}
392 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000393 ${service_type}= Get Variable Value ${src['service_type']} "null"
Hardik Windlassd3f63892021-03-25 08:28:02 +0000394 ${of_id}= Get ofID From OLT List ${src['olt']}
395 ${onu_device_id}= Get Device ID From SN ${src['onu']}
396 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
397 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
398 # Verify ONU state in voltha
399 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
400 ... ENABLED ACTIVE REACHABLE
401 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
402 # Check ONU port is Disabled in ONOS
403 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000404 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000405 # Verify Ping
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000406 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
407 ... Run Keyword And Continue On Failure Check Ping True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000408 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
409 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
410 END
411 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700412 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000413 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
414 ... Running
415 Run Keyword If ${has_dataplane} Clean Up Linux
416 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
417 Log to console Pod ${podName} restarted and sanity checks passed successfully
418
419Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
420 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
421 ... simulate a POD crash. The test then scales the rw-core back to a single instance
422 ... and configures ONOS for access. The test succeeds if the device is able to
423 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000424 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000425 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
426 ... AND Clear All Devices Then Create New Device
427 [Teardown] Run Keywords Collect Logs
428 ... AND Stop Logging RwCoreFailAndRestart-TT
429 Run Keyword If ${has_dataplane} Clean Up Linux
430 FOR ${I} IN RANGE 0 ${olt_count}
431 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
432 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
433 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
434 ... ${olt_serial_number}
435 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
436 ... Get NNI Port in ONOS ${of_id}
437 END
438 FOR ${I} IN RANGE 0 ${num_all_onus}
439 ${src}= Set Variable ${hosts.src[${I}]}
440 ${dst}= Set Variable ${hosts.dst[${I}]}
441 ${of_id}= Get ofID From OLT List ${src['olt']}
442 ${onu_device_id}= Get Device ID From SN ${src['onu']}
443 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
444 ... ${of_id}
445 # Bring up the device and verify it authenticates
446 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000447 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000448 END
449
450 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
451 Scale K8s Deployment voltha voltha-voltha-rw-core 0
452 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
453 # Ensure the ofagent POD goes "not-ready" as expected
454 Wait Until keyword Succeeds ${timeout} 2s
455 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
456 # Scale up the core deployment and make sure both it and the ofagent deployment are back
457 Scale K8s Deployment voltha voltha-voltha-rw-core 1
458 Wait Until Keyword Succeeds ${timeout} 2s
459 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
460 Wait Until Keyword Succeeds ${timeout} 2s
461 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
462 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
463 # so restart the port forwarding for the API service
464 Restart VOLTHA Port Forward voltha-api
465 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
466 # represents system connectivity being restored
467 FOR ${I} IN RANGE 0 ${olt_count}
468 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
469 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
470 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
471 ... ${olt_serial_number}
472 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
473 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
474 END
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000475 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000476
477*** Keywords ***
478Setup Suite
479 [Documentation] Set up the test suite
480 Common Test Suite Setup
481 ${switch_type}= Get Variable Value ${web_power_switch.type}
482 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
483
TorstenThieme136ebf72021-02-26 14:27:07 +0000484Teardown Suite
485 [Documentation] Tear down steps for the suite
486 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass04896f62021-03-05 16:15:11 +0530487 Run Keyword If ${teardown_device} Delete All Devices and Verify
TorstenThieme4e2168e2021-06-22 14:01:47 +0000488 Close All ONOS SSH Connections
Hardik Windlassd3f63892021-03-25 08:28:02 +0000489
490Clear All Devices Then Create New Device
491 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
492 # Remove all devices from voltha and onos
493 Delete All Devices and Verify
494 # Execute normal test Setup Keyword
495 Setup