blob: dc3ac9e95338b8a5877488d88b5ee03d4a3d1202 [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
TorstenThieme136ebf72021-02-26 14:27:07 +000047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053054${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000055${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
TorstenThieme136ebf72021-02-26 14:27:07 +000064${suppressaddsubscriber} True
65
Hardik Windlassb1bda362021-11-24 11:54:36 +000066# flag to choose the subscriber provisioning command type in ONOS
67# TT often provision a single services for a subscriber (eg: hsia, voip, ...) one after the other.
68# if set to True, command used is "volt-add-subscriber-unitag"
69# if set to False, comand used is "volt-add-subscriber-access"
70${unitag_sub} False
71
TorstenThieme136ebf72021-02-26 14:27:07 +000072*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000073Verify ONU after Rebooting Physically for TT
74 [Documentation] Test the ONU functionality by physically turning on/off ONU.
75 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
76 ... Test case runs only on the PODs that are configured with PowerSwitch that
77 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
78 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
79 [Setup] Start Logging RebootOnu_PowerSwitch_TT
80 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +000081 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070082 ... AND Run Keyword If ${logging} Collect Logs
83 ... AND Stop Logging RebootOnu_PowerSwitch_TT
Hardik Windlass364ec452021-04-14 07:12:28 +000084 # Add OLT device
85 Setup
86 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
87 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +000088 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +000089 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
90 Run Keyword If ${has_dataplane} Clean Up Linux
91 FOR ${I} IN RANGE 0 ${num_all_onus}
92 ${src}= Set Variable ${hosts.src[${I}]}
93 ${dst}= Set Variable ${hosts.dst[${I}]}
94 ${service_type}= Get Variable Value ${src['service_type']} "null"
95 ${of_id}= Get ofID From OLT List ${src['olt']}
96 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +000097 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass364ec452021-04-14 07:12:28 +000098 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
99 # Disable Power Switch
100 Disable Switch Outlet ${src['power_switch_port']}
101 # TODO: Add verification for MCAST
102 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000103 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass364ec452021-04-14 07:12:28 +0000104 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
105 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
106 # Remove Subscriber Access (To replicate TT workflow)
Hardik Windlassb1bda362021-11-24 11:54:36 +0000107 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
108 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
109 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
110 ... ELSE
111 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
TorstenThieme4e2168e2021-06-22 14:01:47 +0000112 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000113 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlass364ec452021-04-14 07:12:28 +0000114 Sleep 5s
115 # Enable Power Switch
116 Enable Switch Outlet ${src['power_switch_port']}
117 # Check ONU port is Enabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000118 Wait Until Keyword Succeeds 120s 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000119 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000120 # Add Subscriber Access
121 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
122 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
123 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
124 ... ELSE
125 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000126 Wait Until Keyword Succeeds ${timeout} 2
Hardik Windlassb1bda362021-11-24 11:54:36 +0000127 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass364ec452021-04-14 07:12:28 +0000128 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000129 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass364ec452021-04-14 07:12:28 +0000130 ... ENABLED ACTIVE REACHABLE
131 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
132 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
133 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
134 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
135 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
136 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
137 ... ${dst['container_name']}
138 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000139 ... Sanity Test TT MCAST one ONU ${src}
Hardik Windlass364ec452021-04-14 07:12:28 +0000140 ... ${dst} ${suppressaddsubscriber}
141 END
142
143Verify OLT after Rebooting Physically for TT
144 [Documentation] Test the physical reboot of the OLT
145 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
146 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
147 [Tags] functionalTT PhysicalOltRebootTT
148 [Setup] Start Logging RebootOlt_Physical_TT
149 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +0000150 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700151 ... AND Run Keyword If ${logging} Collect Logs
152 ... AND Stop Logging RebootOlt_Physical_TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000153 # Add OLT device
154 Setup
155 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
156 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000157 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000158 # Reboot the OLT from the OLT CLI
159 FOR ${I} IN RANGE 0 ${olt_count}
160 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
161 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
162 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
163 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
164 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
165 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
166 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
167 END
168 FOR ${I} IN RANGE 0 ${num_all_onus}
169 ${src}= Set Variable ${hosts.src[${I}]}
170 ${dst}= Set Variable ${hosts.dst[${I}]}
171 ${service_type}= Get Variable Value ${src['service_type']} "null"
172 # TODO: Add verification for MCAST
173 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000174 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass364ec452021-04-14 07:12:28 +0000175 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
176 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
177 END
178 # Wait for the OLT to come back up
179 FOR ${I} IN RANGE 0 ${olt_count}
180 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
181 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
182 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
183 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
184 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
185 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
186 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000187 Wait Until Keyword Succeeds 360s 10s
Hardik Windlass364ec452021-04-14 07:12:28 +0000188 ... Validate OLT Device ENABLED ACTIVE
189 ... REACHABLE ${olt_serial_number}
190 END
191 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000192 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000193
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000194Verify ONU Soft Reboot for TT
195 [Documentation] Test the ONU Soft Reboot functionality.
196 [Tags] functionalTT OnuSoftRebootTT
197 [Setup] Start Logging SoftRebootOnu_TT
198 [Teardown] Run Keywords Collect Logs
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000199 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700200 ... AND Collect Logs
201 ... AND Stop Logging SoftRebootOnu_TT
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000202 # Add OLT device
203 Setup
204 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
205 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000206 Perform Sanity Tests TT
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000207 FOR ${I} IN RANGE 0 ${num_all_onus}
208 ${src}= Set Variable ${hosts.src[${I}]}
209 ${dst}= Set Variable ${hosts.dst[${I}]}
210 ${service_type}= Get Variable Value ${src['service_type']} "null"
211 ${of_id}= Get ofID From OLT List ${src['olt']}
212 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000213 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000214 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
215 Reboot Device ${onu_device_id}
216 # TODO: Add verification for MCAST
217 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000218 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000219 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
220 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
221 # Remove Subscriber Access (To replicate TT workflow)
Hardik Windlassb1bda362021-11-24 11:54:36 +0000222 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
223 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
224 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
225 ... ELSE
226 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
TorstenThieme4e2168e2021-06-22 14:01:47 +0000227 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000228 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000229 # Check ONU port is Enabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000230 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000231 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000232 # Add Subscriber Access
233 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
234 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
235 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
236 ... ELSE
237 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000238 Wait Until Keyword Succeeds ${timeout} 2
Hardik Windlassb1bda362021-11-24 11:54:36 +0000239 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000240 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000241 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000242 ... ENABLED ACTIVE REACHABLE
243 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
244 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
245 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
246 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
247 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
248 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
249 ... ${dst['container_name']}
250 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000251 ... Sanity Test TT MCAST one ONU ${src}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000252 ... ${dst} ${suppressaddsubscriber}
253 END
254 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000255 Perform Sanity Tests TT ${suppressaddsubscriber}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000256
Hardik Windlass656661a2021-04-23 03:46:31 +0000257Verify OLT Soft Reboot for TT
258 [Documentation] Test the OLT Soft Reboot functionality.
259 [Tags] functionalTT OltSoftRebootTT
260 [Setup] Start Logging SoftRebootOlt_TT
261 [Teardown] Run Keywords Collect Logs
Hardik Windlass656661a2021-04-23 03:46:31 +0000262 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700263 ... AND Collect Logs
264 ... AND Stop Logging SoftRebootOlt_TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000265 # Add OLT device
266 Setup
267 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
268 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000269 Perform Sanity Tests TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000270 # Reboot the OLT from the OLT CLI
271 FOR ${I} IN RANGE 0 ${olt_count}
272 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
273 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
274 # Reboot the OLT using "voltctl device reboot" command
275 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200276 # Wait for the OLT to actually go down
277 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
278 ... ${olt_serial_number}
Hardik Windlass656661a2021-04-23 03:46:31 +0000279 END
280 FOR ${I} IN RANGE 0 ${num_all_onus}
281 ${src}= Set Variable ${hosts.src[${I}]}
282 ${dst}= Set Variable ${hosts.dst[${I}]}
283 ${service_type}= Get Variable Value ${src['service_type']} "null"
284 # TODO: Add verification for MCAST
285 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000286 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass656661a2021-04-23 03:46:31 +0000287 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
288 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
289 END
290 # Wait for the OLT to come back up
291 FOR ${I} IN RANGE 0 ${olt_count}
292 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
293 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
294 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
295 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
296 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
297 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
298 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000299 Wait Until Keyword Succeeds 360s 10s
Hardik Windlass656661a2021-04-23 03:46:31 +0000300 ... Validate OLT Device ENABLED ACTIVE
301 ... REACHABLE ${olt_serial_number}
302 END
303 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000304 Perform Sanity Tests TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000305
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000306Verify restart openolt-adapter container before subscriber provisioning for TT
307 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
308 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
309 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
310 [Teardown] Run Keywords Collect Logs
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700311 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000312 # Add OLT device
313 Setup
314 Run Keyword If ${has_dataplane} Clean Up Linux
315 FOR ${I} IN RANGE 0 ${olt_count}
316 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
317 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
318 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
319 ... ${olt_serial_number}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000320 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000321 ... Get NNI Port in ONOS ${of_id}
322 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
323 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
324 ... ${nni_port}
325 END
326 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
327 Log ${podStatusOutput}
328 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
329 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
330 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
331 Sleep 5s
332 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
333 ... app ${podName} Running
334 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
335 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
336 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000337 Perform Sanity Tests TT
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000338 Log to console Pod ${podName} restarted and sanity checks passed successfully
339
Hardik Windlassd3f63892021-03-25 08:28:02 +0000340Verify restart ofagent container after subscriber is provisioned for TT
341 [Documentation] Restart ofagent container after VOLTHA is operational.
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000342 [Tags] functionalTT ofagentRestart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000343 [Setup] Start Logging ofagentRestart-TT
344 [Teardown] Run Keywords Collect Logs
345 ... AND Stop Logging ofagentRestart-TT
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700346 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000347 # set timeout value
348 ${waitforRestart} Set Variable 120s
349 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
350 Log ${podStatusOutput}
351 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
352 ${podName} Set Variable ofagent
353 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
354 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
355 ... app ${podName} Running
356 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
357 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000358 Perform Sanity Test TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000359 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
360 Log ${podStatusOutput}
361 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
362 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
363 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700364 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlassd3f63892021-03-25 08:28:02 +0000365 Sleep 30s
366 FOR ${I} IN RANGE 0 ${num_all_onus}
367 ${src}= Set Variable ${hosts.src[${I}]}
368 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000369 ${service_type}= Get Variable Value ${src['service_type']} "null"
Hardik Windlassd3f63892021-03-25 08:28:02 +0000370 ${of_id}= Get ofID From OLT List ${src['olt']}
371 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000372 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd3f63892021-03-25 08:28:02 +0000373 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
374 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000375 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlassd3f63892021-03-25 08:28:02 +0000376 ... ENABLED ACTIVE REACHABLE
377 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
378 # Check ONU port is Disabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000379 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000380 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000381 # Verify Ping
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000382 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000383 ... Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Check Ping True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000384 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
385 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
386 END
387 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700388 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000389 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
390 ... Running
391 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000392 Perform Sanity Tests TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000393 Log to console Pod ${podName} restarted and sanity checks passed successfully
394
395Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
396 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
397 ... simulate a POD crash. The test then scales the rw-core back to a single instance
398 ... and configures ONOS for access. The test succeeds if the device is able to
399 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000400 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000401 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
402 ... AND Clear All Devices Then Create New Device
403 [Teardown] Run Keywords Collect Logs
404 ... AND Stop Logging RwCoreFailAndRestart-TT
405 Run Keyword If ${has_dataplane} Clean Up Linux
406 FOR ${I} IN RANGE 0 ${olt_count}
407 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
408 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
409 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
410 ... ${olt_serial_number}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000411 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd3f63892021-03-25 08:28:02 +0000412 ... Get NNI Port in ONOS ${of_id}
413 END
414 FOR ${I} IN RANGE 0 ${num_all_onus}
415 ${src}= Set Variable ${hosts.src[${I}]}
416 ${dst}= Set Variable ${hosts.dst[${I}]}
417 ${of_id}= Get ofID From OLT List ${src['olt']}
418 ${onu_device_id}= Get Device ID From SN ${src['onu']}
419 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
420 ... ${of_id}
421 # Bring up the device and verify it authenticates
422 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000423 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000424 END
425
426 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
427 Scale K8s Deployment voltha voltha-voltha-rw-core 0
428 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
429 # Ensure the ofagent POD goes "not-ready" as expected
430 Wait Until keyword Succeeds ${timeout} 2s
431 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
432 # Scale up the core deployment and make sure both it and the ofagent deployment are back
433 Scale K8s Deployment voltha voltha-voltha-rw-core 1
434 Wait Until Keyword Succeeds ${timeout} 2s
435 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
436 Wait Until Keyword Succeeds ${timeout} 2s
437 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
438 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
439 # so restart the port forwarding for the API service
440 Restart VOLTHA Port Forward voltha-api
441 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
442 # represents system connectivity being restored
443 FOR ${I} IN RANGE 0 ${olt_count}
444 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
445 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
446 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
447 ... ${olt_serial_number}
448 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
449 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
450 END
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000451 Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000452
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000453Verify restart openonu-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000454 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
455 ... Run the ping continuously in background during container restart,
456 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000457 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000458 [Tags] functionalTT Restart-OpenOnu-Ping-TT dataplaneTT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000459 [Setup] Start Logging Restart-OpenOnu-Ping-TT
460 [Teardown] Run Keywords Collect Logs
461 ... AND Stop Logging Restart-OpenOnu-Ping-TT
462 Clear All Devices Then Create New Device
463 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
464 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000465 Perform Sanity Test TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000466 FOR ${I} IN RANGE 0 ${num_all_onus}
467 ${src}= Set Variable ${hosts.src[${I}]}
468 ${dst}= Set Variable ${hosts.dst[${I}]}
469 ${service_type}= Get Variable Value ${src['service_type']} "null"
470 Continue For Loop If '${service_type}' == 'mcast'
471 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
472 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000473 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000474 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
475 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
476 END
477 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
478 Log ${podStatusOutput}
479 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
480 ${podName} Set Variable adapter-open-onu
481 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
482 Sleep 5s
483 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
484 ... app ${podName} Running
485 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
486 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
487 Log ${podStatusOutput}
488 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
489 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
490 FOR ${I} IN RANGE 0 ${num_all_onus}
491 ${src}= Set Variable ${hosts.src[${I}]}
492 ${dst}= Set Variable ${hosts.dst[${I}]}
493 ${service_type}= Get Variable Value ${src['service_type']} "null"
494 Continue For Loop If '${service_type}' == 'mcast'
495 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000496 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000497 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
498 ... ${src['container_type']} ${src['container_name']}
499 END
500 FOR ${I} IN RANGE 0 ${num_all_onus}
501 ${src}= Set Variable ${hosts.src[${I}]}
502 ${dst}= Set Variable ${hosts.dst[${I}]}
503 ${service_type}= Get Variable Value ${src['service_type']} "null"
504 Continue For Loop If '${service_type}' == 'mcast'
505 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
506 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000507 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000508 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
509 ... ${src['container_type']} ${src['container_name']}
510 Check Ping Result True ${ping_output}
511 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000512 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
513 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000514
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000515Verify restart openolt-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000516 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
517 ... Run the ping continuously in background during container restart,
518 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000519 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000520 [Tags] functionalTT Restart-OpenOlt-Ping-TT dataplaneTT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000521 [Setup] Start Logging Restart-OpenOlt-Ping-TT
522 [Teardown] Run Keywords Collect Logs
523 ... AND Stop Logging Restart-OpenOlt-Ping-TT
524 Clear All Devices Then Create New Device
525 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
526 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000527 Perform Sanity Test TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000528 FOR ${I} IN RANGE 0 ${num_all_onus}
529 ${src}= Set Variable ${hosts.src[${I}]}
530 ${dst}= Set Variable ${hosts.dst[${I}]}
531 ${service_type}= Get Variable Value ${src['service_type']} "null"
532 Continue For Loop If '${service_type}' == 'mcast'
533 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
534 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000535 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000536 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
537 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
538 END
539 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
540 Log ${podStatusOutput}
541 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
542 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
543 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
544 Sleep 5s
545 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
546 ... app ${podName} Running
547 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
548 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
549 Log ${podStatusOutput}
550 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
551 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
552 FOR ${I} IN RANGE 0 ${num_all_onus}
553 ${src}= Set Variable ${hosts.src[${I}]}
554 ${dst}= Set Variable ${hosts.dst[${I}]}
555 ${service_type}= Get Variable Value ${src['service_type']} "null"
556 Continue For Loop If '${service_type}' == 'mcast'
557 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000558 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000559 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
560 ... ${src['container_type']} ${src['container_name']}
561 END
562 FOR ${I} IN RANGE 0 ${num_all_onus}
563 ${src}= Set Variable ${hosts.src[${I}]}
564 ${dst}= Set Variable ${hosts.dst[${I}]}
565 ${service_type}= Get Variable Value ${src['service_type']} "null"
566 Continue For Loop If '${service_type}' == 'mcast'
567 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
568 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000569 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000570 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
571 ... ${src['container_type']} ${src['container_name']}
572 Check Ping Result True ${ping_output}
573 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000574 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
575 Verify Control Plane After Pod Restart TT
576
577Verify restart rw-core container for TT
578 [Documentation] Restart rw-core container after VOLTHA is operational.
579 ... Run the ping continuously in background during container restart,
580 ... and verify that there should be no affect on the dataplane.
581 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000582 [Tags] functionalTT Restart-RwCore-Ping-TT dataplaneTT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000583 [Setup] Start Logging Restart-RwCore-Ping-TT
584 [Teardown] Run Keywords Collect Logs
585 ... AND Stop Logging Restart-RwCore-Ping-TT
586 Clear All Devices Then Create New Device
587 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
588 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000589 Perform Sanity Test TT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000590 FOR ${I} IN RANGE 0 ${num_all_onus}
591 ${src}= Set Variable ${hosts.src[${I}]}
592 ${dst}= Set Variable ${hosts.dst[${I}]}
593 ${service_type}= Get Variable Value ${src['service_type']} "null"
594 Continue For Loop If '${service_type}' == 'mcast'
595 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
596 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000597 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000598 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
599 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
600 END
601 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
602 Log ${podStatusOutput}
603 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
604 ${podName} Set Variable rw-core
605 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
606 Sleep 5s
607 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
608 ... app ${podName} Running
609 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
610 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
611 # so restart the port forwarding for the API service
612 Restart VOLTHA Port Forward voltha-api
613 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
614 Log ${podStatusOutput}
615 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
616 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
617 FOR ${I} IN RANGE 0 ${num_all_onus}
618 ${src}= Set Variable ${hosts.src[${I}]}
619 ${dst}= Set Variable ${hosts.dst[${I}]}
620 ${service_type}= Get Variable Value ${src['service_type']} "null"
621 Continue For Loop If '${service_type}' == 'mcast'
622 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000623 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000624 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
625 ... ${src['container_type']} ${src['container_name']}
626 END
627 FOR ${I} IN RANGE 0 ${num_all_onus}
628 ${src}= Set Variable ${hosts.src[${I}]}
629 ${dst}= Set Variable ${hosts.dst[${I}]}
630 ${service_type}= Get Variable Value ${src['service_type']} "null"
631 Continue For Loop If '${service_type}' == 'mcast'
632 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
633 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000634 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000635 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
636 ... ${src['container_type']} ${src['container_name']}
637 Check Ping Result True ${ping_output}
638 END
639 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
640 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000641
TorstenThieme136ebf72021-02-26 14:27:07 +0000642*** Keywords ***
643Setup Suite
644 [Documentation] Set up the test suite
645 Common Test Suite Setup
646 ${switch_type}= Get Variable Value ${web_power_switch.type}
647 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
648
Hardik Windlassd3f63892021-03-25 08:28:02 +0000649Clear All Devices Then Create New Device
650 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
651 # Remove all devices from voltha and onos
652 Delete All Devices and Verify
653 # Execute normal test Setup Keyword
654 Setup
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000655
656Verify Control Plane After Pod Restart TT
657 [Documentation] Verifies the control plane functionality after the voltha pod restart
658 ... by deleting and re-adding the subscriber
659 Run Keyword If ${has_dataplane} Clean Up Linux
660 FOR ${I} IN RANGE 0 ${num_all_onus}
661 ${src}= Set Variable ${hosts.src[${I}]}
662 ${dst}= Set Variable ${hosts.dst[${I}]}
663 ${service_type}= Get Variable Value ${src['service_type']} "null"
664 ${of_id}= Get ofID From OLT List ${src['olt']}
665 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
666 ... ${of_id} ${src['uni_id']}
667 # Remove Subscriber Access
Hardik Windlassb1bda362021-11-24 11:54:36 +0000668 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
669 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
670 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
671 ... ELSE
672 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000673 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000674 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000675 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000676 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000677 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
678 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
679 # Add Subscriber Access
Hardik Windlassb1bda362021-11-24 11:54:36 +0000680 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
681 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
682 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
683 ... ELSE
Matteo Scandolo2786d212021-12-06 12:22:53 -0800684 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000685 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000686 ... ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000687 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000688 ... Validate Device ENABLED ACTIVE
689 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass5a2d1f42021-11-12 12:52:04 +0000690 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000691 ... Sanity Test TT one ONU ${src} ${dst} ${suppressaddsubscriber}
Hardik Windlass5a2d1f42021-11-12 12:52:04 +0000692 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000693 ... Sanity Test TT MCAST one ONU ${src} ${dst} ${suppressaddsubscriber}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000694 END