blob: 976a75c5584767c0d795619e2898548671d06190 [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
TorstenThieme136ebf72021-02-26 14:27:07 +000047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053054${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000055${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
TorstenThieme136ebf72021-02-26 14:27:07 +000064${suppressaddsubscriber} True
65
Hardik Windlassb1bda362021-11-24 11:54:36 +000066# flag to choose the subscriber provisioning command type in ONOS
67# TT often provision a single services for a subscriber (eg: hsia, voip, ...) one after the other.
68# if set to True, command used is "volt-add-subscriber-unitag"
69# if set to False, comand used is "volt-add-subscriber-access"
70${unitag_sub} False
71
TorstenThieme136ebf72021-02-26 14:27:07 +000072*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000073Verify ONU after Rebooting Physically for TT
74 [Documentation] Test the ONU functionality by physically turning on/off ONU.
75 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
76 ... Test case runs only on the PODs that are configured with PowerSwitch that
77 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
78 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
79 [Setup] Start Logging RebootOnu_PowerSwitch_TT
80 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +000081 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070082 ... AND Run Keyword If ${logging} Collect Logs
83 ... AND Stop Logging RebootOnu_PowerSwitch_TT
Hardik Windlass364ec452021-04-14 07:12:28 +000084 # Add OLT device
85 Setup
86 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
87 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +000088 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +000089 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
90 Run Keyword If ${has_dataplane} Clean Up Linux
91 FOR ${I} IN RANGE 0 ${num_all_onus}
92 ${src}= Set Variable ${hosts.src[${I}]}
93 ${dst}= Set Variable ${hosts.dst[${I}]}
94 ${service_type}= Get Variable Value ${src['service_type']} "null"
95 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +000096 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass364ec452021-04-14 07:12:28 +000097 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +053098 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Girish Gowdraa3a05c62022-02-16 13:54:37 -080099 # If the power switch port is not specified, continue
100 Continue For Loop If '${src["power_switch_port"]}' == '${None}'
Hardik Windlass364ec452021-04-14 07:12:28 +0000101 # Disable Power Switch
102 Disable Switch Outlet ${src['power_switch_port']}
103 # TODO: Add verification for MCAST
104 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000105 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass364ec452021-04-14 07:12:28 +0000106 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
107 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
108 # Remove Subscriber Access (To replicate TT workflow)
Hardik Windlassb1bda362021-11-24 11:54:36 +0000109 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
110 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
111 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
112 ... ELSE
113 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
TorstenThieme4e2168e2021-06-22 14:01:47 +0000114 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000115 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlass364ec452021-04-14 07:12:28 +0000116 Sleep 5s
117 # Enable Power Switch
118 Enable Switch Outlet ${src['power_switch_port']}
119 # Check ONU port is Enabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000120 Wait Until Keyword Succeeds 120s 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000121 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000122 # Add Subscriber Access
123 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
124 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
125 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
126 ... ELSE
127 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000128 Wait Until Keyword Succeeds ${timeout} 2
Hardik Windlassb1bda362021-11-24 11:54:36 +0000129 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass364ec452021-04-14 07:12:28 +0000130 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000131 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass364ec452021-04-14 07:12:28 +0000132 ... ENABLED ACTIVE REACHABLE
133 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
134 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
135 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
136 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
137 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
138 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
139 ... ${dst['container_name']}
140 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000141 ... Sanity Test TT MCAST one ONU ${src}
Hardik Windlass364ec452021-04-14 07:12:28 +0000142 ... ${dst} ${suppressaddsubscriber}
143 END
144
145Verify OLT after Rebooting Physically for TT
146 [Documentation] Test the physical reboot of the OLT
147 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
148 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
149 [Tags] functionalTT PhysicalOltRebootTT
150 [Setup] Start Logging RebootOlt_Physical_TT
151 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +0000152 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700153 ... AND Run Keyword If ${logging} Collect Logs
154 ... AND Stop Logging RebootOlt_Physical_TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000155 # Add OLT device
156 Setup
157 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
158 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000159 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000160 # Reboot the OLT from the OLT CLI
161 FOR ${I} IN RANGE 0 ${olt_count}
162 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
163 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
164 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
165 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
166 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
167 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
168 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
169 END
170 FOR ${I} IN RANGE 0 ${num_all_onus}
171 ${src}= Set Variable ${hosts.src[${I}]}
172 ${dst}= Set Variable ${hosts.dst[${I}]}
173 ${service_type}= Get Variable Value ${src['service_type']} "null"
174 # TODO: Add verification for MCAST
175 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000176 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass364ec452021-04-14 07:12:28 +0000177 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
178 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
179 END
180 # Wait for the OLT to come back up
181 FOR ${I} IN RANGE 0 ${olt_count}
182 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
183 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
184 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
185 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
186 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
187 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
188 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000189 Wait Until Keyword Succeeds 360s 10s
Hardik Windlass364ec452021-04-14 07:12:28 +0000190 ... Validate OLT Device ENABLED ACTIVE
191 ... REACHABLE ${olt_serial_number}
192 END
193 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000194 Perform Sanity Tests TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000195
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000196Verify ONU Soft Reboot for TT
197 [Documentation] Test the ONU Soft Reboot functionality.
198 [Tags] functionalTT OnuSoftRebootTT
199 [Setup] Start Logging SoftRebootOnu_TT
200 [Teardown] Run Keywords Collect Logs
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000201 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700202 ... AND Collect Logs
203 ... AND Stop Logging SoftRebootOnu_TT
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000204 # Add OLT device
205 Setup
206 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
207 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000208 Perform Sanity Tests TT
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000209 FOR ${I} IN RANGE 0 ${num_all_onus}
210 ${src}= Set Variable ${hosts.src[${I}]}
211 ${dst}= Set Variable ${hosts.dst[${I}]}
212 ${service_type}= Get Variable Value ${src['service_type']} "null"
213 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000214 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000215 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530216 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000217 Reboot Device ${onu_device_id}
218 # TODO: Add verification for MCAST
219 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000220 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000221 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
222 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
223 # Remove Subscriber Access (To replicate TT workflow)
Hardik Windlassb1bda362021-11-24 11:54:36 +0000224 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
225 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
226 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
227 ... ELSE
228 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
TorstenThieme4e2168e2021-06-22 14:01:47 +0000229 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000230 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000231 # Check ONU port is Enabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000232 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000233 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000234 # Add Subscriber Access
235 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
236 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
237 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
238 ... ELSE
239 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000240 Wait Until Keyword Succeeds ${timeout} 2
Hardik Windlassb1bda362021-11-24 11:54:36 +0000241 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000242 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000243 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000244 ... ENABLED ACTIVE REACHABLE
245 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
246 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
247 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
248 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
249 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
250 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
251 ... ${dst['container_name']}
252 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000253 ... Sanity Test TT MCAST one ONU ${src}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000254 ... ${dst} ${suppressaddsubscriber}
255 END
256 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000257 Perform Sanity Tests TT ${suppressaddsubscriber}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000258
Hardik Windlass656661a2021-04-23 03:46:31 +0000259Verify OLT Soft Reboot for TT
260 [Documentation] Test the OLT Soft Reboot functionality.
261 [Tags] functionalTT OltSoftRebootTT
262 [Setup] Start Logging SoftRebootOlt_TT
263 [Teardown] Run Keywords Collect Logs
Hardik Windlass656661a2021-04-23 03:46:31 +0000264 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700265 ... AND Collect Logs
266 ... AND Stop Logging SoftRebootOlt_TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000267 # Add OLT device
268 Setup
269 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
270 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000271 Perform Sanity Tests TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000272 # Reboot the OLT from the OLT CLI
273 FOR ${I} IN RANGE 0 ${olt_count}
274 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
275 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
276 # Reboot the OLT using "voltctl device reboot" command
277 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200278 # Wait for the OLT to actually go down
279 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
280 ... ${olt_serial_number}
Hardik Windlass656661a2021-04-23 03:46:31 +0000281 END
282 FOR ${I} IN RANGE 0 ${num_all_onus}
283 ${src}= Set Variable ${hosts.src[${I}]}
284 ${dst}= Set Variable ${hosts.dst[${I}]}
285 ${service_type}= Get Variable Value ${src['service_type']} "null"
286 # TODO: Add verification for MCAST
287 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000288 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass656661a2021-04-23 03:46:31 +0000289 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
290 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
291 END
292 # Wait for the OLT to come back up
293 FOR ${I} IN RANGE 0 ${olt_count}
294 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
295 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
296 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
297 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
298 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
299 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
300 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000301 Wait Until Keyword Succeeds 360s 10s
Hardik Windlass656661a2021-04-23 03:46:31 +0000302 ... Validate OLT Device ENABLED ACTIVE
303 ... REACHABLE ${olt_serial_number}
304 END
305 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000306 Perform Sanity Tests TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000307
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000308Verify restart openolt-adapter container before subscriber provisioning for TT
309 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
310 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
311 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
312 [Teardown] Run Keywords Collect Logs
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700313 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000314 # Add OLT device
315 Setup
316 Run Keyword If ${has_dataplane} Clean Up Linux
317 FOR ${I} IN RANGE 0 ${olt_count}
318 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
319 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
320 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
321 ... ${olt_serial_number}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000322 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000323 ... Get NNI Port in ONOS ${of_id}
324 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
325 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
326 ... ${nni_port}
327 END
328 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
329 Log ${podStatusOutput}
330 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
331 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
332 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
333 Sleep 5s
334 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
335 ... app ${podName} Running
336 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
337 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
338 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000339 Perform Sanity Tests TT
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000340 Log to console Pod ${podName} restarted and sanity checks passed successfully
341
Hardik Windlassd3f63892021-03-25 08:28:02 +0000342Verify restart ofagent container after subscriber is provisioned for TT
343 [Documentation] Restart ofagent container after VOLTHA is operational.
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000344 [Tags] functionalTT ofagentRestart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000345 [Setup] Start Logging ofagentRestart-TT
346 [Teardown] Run Keywords Collect Logs
347 ... AND Stop Logging ofagentRestart-TT
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700348 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530349 ... AND Wait Until Keyword Succeeds ${timeout} 2s
350 ... Validate Pods Status By Label ${NAMESPACE} app ofagent Running
351 ... AND Wait Until Keyword Succeeds ${timeout} 3s
352 ... Pods Are Ready By Label ${NAMESPACE} app ofagent
Hardik Windlassd3f63892021-03-25 08:28:02 +0000353 # set timeout value
354 ${waitforRestart} Set Variable 120s
355 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
356 Log ${podStatusOutput}
357 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
358 ${podName} Set Variable ofagent
359 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
360 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
361 ... app ${podName} Running
362 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
363 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000364 Perform Sanity Test TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000365 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
366 Log ${podStatusOutput}
367 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
368 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
369 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700370 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlassd3f63892021-03-25 08:28:02 +0000371 Sleep 30s
372 FOR ${I} IN RANGE 0 ${num_all_onus}
373 ${src}= Set Variable ${hosts.src[${I}]}
374 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000375 ${service_type}= Get Variable Value ${src['service_type']} "null"
Hardik Windlassd3f63892021-03-25 08:28:02 +0000376 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000377 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd3f63892021-03-25 08:28:02 +0000378 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530379 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000380 # Verify ONU state in voltha
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000381 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlassd3f63892021-03-25 08:28:02 +0000382 ... ENABLED ACTIVE REACHABLE
383 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
384 # Check ONU port is Disabled in ONOS
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000385 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000386 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000387 # Verify Ping
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000388 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000389 ... Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Check Ping True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000390 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
391 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
392 END
393 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700394 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000395 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
396 ... Running
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530397 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000398 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000399 Perform Sanity Tests TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000400 Log to console Pod ${podName} restarted and sanity checks passed successfully
401
402Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
403 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
404 ... simulate a POD crash. The test then scales the rw-core back to a single instance
405 ... and configures ONOS for access. The test succeeds if the device is able to
406 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000407 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000408 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
409 ... AND Clear All Devices Then Create New Device
410 [Teardown] Run Keywords Collect Logs
411 ... AND Stop Logging RwCoreFailAndRestart-TT
412 Run Keyword If ${has_dataplane} Clean Up Linux
413 FOR ${I} IN RANGE 0 ${olt_count}
414 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
415 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
416 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
417 ... ${olt_serial_number}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000418 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd3f63892021-03-25 08:28:02 +0000419 ... Get NNI Port in ONOS ${of_id}
420 END
421 FOR ${I} IN RANGE 0 ${num_all_onus}
422 ${src}= Set Variable ${hosts.src[${I}]}
423 ${dst}= Set Variable ${hosts.dst[${I}]}
424 ${of_id}= Get ofID From OLT List ${src['olt']}
425 ${onu_device_id}= Get Device ID From SN ${src['onu']}
426 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
427 ... ${of_id}
428 # Bring up the device and verify it authenticates
429 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000430 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000431 END
432
433 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
434 Scale K8s Deployment voltha voltha-voltha-rw-core 0
435 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
436 # Ensure the ofagent POD goes "not-ready" as expected
437 Wait Until keyword Succeeds ${timeout} 2s
438 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
439 # Scale up the core deployment and make sure both it and the ofagent deployment are back
440 Scale K8s Deployment voltha voltha-voltha-rw-core 1
441 Wait Until Keyword Succeeds ${timeout} 2s
442 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
443 Wait Until Keyword Succeeds ${timeout} 2s
444 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
445 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
446 # so restart the port forwarding for the API service
447 Restart VOLTHA Port Forward voltha-api
448 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
449 # represents system connectivity being restored
450 FOR ${I} IN RANGE 0 ${olt_count}
451 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
452 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
453 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
454 ... ${olt_serial_number}
455 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
456 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
457 END
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000458 Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000459
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000460Verify restart openonu-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000461 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
462 ... Run the ping continuously in background during container restart,
463 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000464 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000465 [Tags] functionalTT Restart-OpenOnu-Ping-TT dataplaneTT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000466 [Setup] Start Logging Restart-OpenOnu-Ping-TT
467 [Teardown] Run Keywords Collect Logs
468 ... AND Stop Logging Restart-OpenOnu-Ping-TT
469 Clear All Devices Then Create New Device
470 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
471 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000472 Perform Sanity Test TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000473 FOR ${I} IN RANGE 0 ${num_all_onus}
474 ${src}= Set Variable ${hosts.src[${I}]}
475 ${dst}= Set Variable ${hosts.dst[${I}]}
476 ${service_type}= Get Variable Value ${src['service_type']} "null"
477 Continue For Loop If '${service_type}' == 'mcast'
478 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
479 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000480 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000481 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
482 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
483 END
484 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
485 Log ${podStatusOutput}
486 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
487 ${podName} Set Variable adapter-open-onu
488 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
489 Sleep 5s
490 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
491 ... app ${podName} Running
492 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
493 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
494 Log ${podStatusOutput}
495 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
496 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
497 FOR ${I} IN RANGE 0 ${num_all_onus}
498 ${src}= Set Variable ${hosts.src[${I}]}
499 ${dst}= Set Variable ${hosts.dst[${I}]}
500 ${service_type}= Get Variable Value ${src['service_type']} "null"
501 Continue For Loop If '${service_type}' == 'mcast'
502 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000503 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000504 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
505 ... ${src['container_type']} ${src['container_name']}
506 END
507 FOR ${I} IN RANGE 0 ${num_all_onus}
508 ${src}= Set Variable ${hosts.src[${I}]}
509 ${dst}= Set Variable ${hosts.dst[${I}]}
510 ${service_type}= Get Variable Value ${src['service_type']} "null"
511 Continue For Loop If '${service_type}' == 'mcast'
512 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
513 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000514 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000515 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
516 ... ${src['container_type']} ${src['container_name']}
517 Check Ping Result True ${ping_output}
518 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000519 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
520 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000521
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000522Verify restart openolt-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000523 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
524 ... Run the ping continuously in background during container restart,
525 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000526 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000527 [Tags] functionalTT Restart-OpenOlt-Ping-TT dataplaneTT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000528 [Setup] Start Logging Restart-OpenOlt-Ping-TT
529 [Teardown] Run Keywords Collect Logs
530 ... AND Stop Logging Restart-OpenOlt-Ping-TT
531 Clear All Devices Then Create New Device
532 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
533 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000534 Perform Sanity Test TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000535 FOR ${I} IN RANGE 0 ${num_all_onus}
536 ${src}= Set Variable ${hosts.src[${I}]}
537 ${dst}= Set Variable ${hosts.dst[${I}]}
538 ${service_type}= Get Variable Value ${src['service_type']} "null"
539 Continue For Loop If '${service_type}' == 'mcast'
540 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
541 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000542 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000543 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
544 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
545 END
546 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
547 Log ${podStatusOutput}
548 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
549 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
550 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
551 Sleep 5s
552 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
553 ... app ${podName} Running
554 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
555 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
556 Log ${podStatusOutput}
557 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
558 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
559 FOR ${I} IN RANGE 0 ${num_all_onus}
560 ${src}= Set Variable ${hosts.src[${I}]}
561 ${dst}= Set Variable ${hosts.dst[${I}]}
562 ${service_type}= Get Variable Value ${src['service_type']} "null"
563 Continue For Loop If '${service_type}' == 'mcast'
564 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000565 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000566 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
567 ... ${src['container_type']} ${src['container_name']}
568 END
569 FOR ${I} IN RANGE 0 ${num_all_onus}
570 ${src}= Set Variable ${hosts.src[${I}]}
571 ${dst}= Set Variable ${hosts.dst[${I}]}
572 ${service_type}= Get Variable Value ${src['service_type']} "null"
573 Continue For Loop If '${service_type}' == 'mcast'
574 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
575 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000576 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd50b0202021-09-02 14:45:09 +0000577 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
578 ... ${src['container_type']} ${src['container_name']}
579 Check Ping Result True ${ping_output}
580 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000581 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
582 Verify Control Plane After Pod Restart TT
583
584Verify restart rw-core container for TT
585 [Documentation] Restart rw-core container after VOLTHA is operational.
586 ... Run the ping continuously in background during container restart,
587 ... and verify that there should be no affect on the dataplane.
588 ... Also, verify that the voltha control plane functionality is not affected.
Hardik Windlassb1bda362021-11-24 11:54:36 +0000589 [Tags] functionalTT Restart-RwCore-Ping-TT dataplaneTT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000590 [Setup] Start Logging Restart-RwCore-Ping-TT
591 [Teardown] Run Keywords Collect Logs
592 ... AND Stop Logging Restart-RwCore-Ping-TT
593 Clear All Devices Then Create New Device
594 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
595 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000596 Perform Sanity Test TT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000597 FOR ${I} IN RANGE 0 ${num_all_onus}
598 ${src}= Set Variable ${hosts.src[${I}]}
599 ${dst}= Set Variable ${hosts.dst[${I}]}
600 ${service_type}= Get Variable Value ${src['service_type']} "null"
601 Continue For Loop If '${service_type}' == 'mcast'
602 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
603 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000604 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000605 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
606 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
607 END
608 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
609 Log ${podStatusOutput}
610 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
611 ${podName} Set Variable rw-core
612 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
613 Sleep 5s
614 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
615 ... app ${podName} Running
616 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
617 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
618 # so restart the port forwarding for the API service
619 Restart VOLTHA Port Forward voltha-api
620 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
621 Log ${podStatusOutput}
622 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
623 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
624 FOR ${I} IN RANGE 0 ${num_all_onus}
625 ${src}= Set Variable ${hosts.src[${I}]}
626 ${dst}= Set Variable ${hosts.dst[${I}]}
627 ${service_type}= Get Variable Value ${src['service_type']} "null"
628 Continue For Loop If '${service_type}' == 'mcast'
629 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000630 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000631 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
632 ... ${src['container_type']} ${src['container_name']}
633 END
634 FOR ${I} IN RANGE 0 ${num_all_onus}
635 ${src}= Set Variable ${hosts.src[${I}]}
636 ${dst}= Set Variable ${hosts.dst[${I}]}
637 ${service_type}= Get Variable Value ${src['service_type']} "null"
638 Continue For Loop If '${service_type}' == 'mcast'
639 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
640 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000641 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000642 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
643 ... ${src['container_type']} ${src['container_name']}
644 Check Ping Result True ${ping_output}
645 END
646 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
647 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000648
TorstenThieme136ebf72021-02-26 14:27:07 +0000649*** Keywords ***
650Setup Suite
651 [Documentation] Set up the test suite
652 Common Test Suite Setup
653 ${switch_type}= Get Variable Value ${web_power_switch.type}
654 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
655
Hardik Windlassd3f63892021-03-25 08:28:02 +0000656Clear All Devices Then Create New Device
657 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
658 # Remove all devices from voltha and onos
659 Delete All Devices and Verify
660 # Execute normal test Setup Keyword
661 Setup
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000662
663Verify Control Plane After Pod Restart TT
664 [Documentation] Verifies the control plane functionality after the voltha pod restart
665 ... by deleting and re-adding the subscriber
666 Run Keyword If ${has_dataplane} Clean Up Linux
667 FOR ${I} IN RANGE 0 ${num_all_onus}
668 ${src}= Set Variable ${hosts.src[${I}]}
669 ${dst}= Set Variable ${hosts.dst[${I}]}
670 ${service_type}= Get Variable Value ${src['service_type']} "null"
671 ${of_id}= Get ofID From OLT List ${src['olt']}
672 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
673 ... ${of_id} ${src['uni_id']}
674 # Remove Subscriber Access
Hardik Windlassb1bda362021-11-24 11:54:36 +0000675 ${del_sub_cmd}= Run Keyword If ${unitag_sub}
676 ... Catenate volt-remove-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
677 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
678 ... ELSE
679 ... Set Variable volt-remove-subscriber-access ${of_id} ${onu_port}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000680 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000681 ... ${ONOS_SSH_PORT} ${del_sub_cmd}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000682 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000683 ... Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000684 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
685 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
686 # Add Subscriber Access
Hardik Windlassb1bda362021-11-24 11:54:36 +0000687 ${add_sub_cmd}= Run Keyword If ${unitag_sub}
688 ... Catenate volt-add-subscriber-unitag --tpId ${src['tp_id']} --sTag ${src['s_tag']}
689 ... --cTag ${src['c_tag']} ${src['onu']}-${src['uni_id']}
690 ... ELSE
Matteo Scandolo2786d212021-12-06 12:22:53 -0800691 ... Set Variable volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000692 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlassb1bda362021-11-24 11:54:36 +0000693 ... ${ONOS_SSH_PORT} ${add_sub_cmd}
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000694 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000695 ... Validate Device ENABLED ACTIVE
696 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass5a2d1f42021-11-12 12:52:04 +0000697 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000698 ... Sanity Test TT one ONU ${src} ${dst} ${suppressaddsubscriber}
Hardik Windlass5a2d1f42021-11-12 12:52:04 +0000699 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast'
Hardik Windlass5083b3a2021-11-11 12:16:22 +0000700 ... Sanity Test TT MCAST one ONU ${src} ${dst} ${suppressaddsubscriber}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000701 END