blob: 392565d63737b622bf634185e304fdb83c5bc671 [file] [log] [blame]
TorstenThieme136ebf72021-02-26 14:27:07 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various functional end-to-end scenarios for TT workflow
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
TorstenThieme136ebf72021-02-26 14:27:07 +000047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
Hardik Windlass04896f62021-03-05 16:15:11 +053054${teardown_device} True
TorstenThieme136ebf72021-02-26 14:27:07 +000055${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
TorstenThieme136ebf72021-02-26 14:27:07 +000064${suppressaddsubscriber} True
65
66*** Test Cases ***
Hardik Windlass364ec452021-04-14 07:12:28 +000067Verify ONU after Rebooting Physically for TT
68 [Documentation] Test the ONU functionality by physically turning on/off ONU.
69 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
70 ... Test case runs only on the PODs that are configured with PowerSwitch that
71 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
72 [Tags] functionalTT PowerSwitchOnuRebootTT PowerSwitch
73 [Setup] Start Logging RebootOnu_PowerSwitch_TT
74 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +000075 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070076 ... AND Run Keyword If ${logging} Collect Logs
77 ... AND Stop Logging RebootOnu_PowerSwitch_TT
Hardik Windlass364ec452021-04-14 07:12:28 +000078 # Add OLT device
79 Setup
80 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
81 Run Keyword If ${has_dataplane} Clean Up Linux
82 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
83 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
84 Run Keyword If ${has_dataplane} Clean Up Linux
85 FOR ${I} IN RANGE 0 ${num_all_onus}
86 ${src}= Set Variable ${hosts.src[${I}]}
87 ${dst}= Set Variable ${hosts.dst[${I}]}
88 ${service_type}= Get Variable Value ${src['service_type']} "null"
89 ${of_id}= Get ofID From OLT List ${src['olt']}
90 ${onu_device_id}= Get Device ID From SN ${src['onu']}
91 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
92 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
93 # Disable Power Switch
94 Disable Switch Outlet ${src['power_switch_port']}
95 # TODO: Add verification for MCAST
96 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
97 ... Wait Until Keyword Succeeds 60s 2s
98 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
99 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
100 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000101 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass364ec452021-04-14 07:12:28 +0000102 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
103 Sleep 5s
104 # Enable Power Switch
105 Enable Switch Outlet ${src['power_switch_port']}
106 # Check ONU port is Enabled in ONOS
107 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000108 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass364ec452021-04-14 07:12:28 +0000109 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000110 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass364ec452021-04-14 07:12:28 +0000111 ... volt-add-subscriber-access ${of_id} ${onu_port}
112 # Verify ONU state in voltha
113 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
114 ... ENABLED ACTIVE REACHABLE
115 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
116 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
117 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
118 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
119 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
120 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
121 ... ${dst['container_name']}
122 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
123 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
124 ... ${dst} ${suppressaddsubscriber}
125 END
126
127Verify OLT after Rebooting Physically for TT
128 [Documentation] Test the physical reboot of the OLT
129 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityTT test was executed)
130 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
131 [Tags] functionalTT PhysicalOltRebootTT
132 [Setup] Start Logging RebootOlt_Physical_TT
133 [Teardown] Run Keywords Collect Logs
Hardik Windlass364ec452021-04-14 07:12:28 +0000134 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700135 ... AND Run Keyword If ${logging} Collect Logs
136 ... AND Stop Logging RebootOlt_Physical_TT
Hardik Windlass364ec452021-04-14 07:12:28 +0000137 # Add OLT device
138 Setup
139 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
140 Run Keyword If ${has_dataplane} Clean Up Linux
141 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
142 # Reboot the OLT from the OLT CLI
143 FOR ${I} IN RANGE 0 ${olt_count}
144 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
145 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
146 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
147 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
148 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
149 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
150 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
151 END
152 FOR ${I} IN RANGE 0 ${num_all_onus}
153 ${src}= Set Variable ${hosts.src[${I}]}
154 ${dst}= Set Variable ${hosts.dst[${I}]}
155 ${service_type}= Get Variable Value ${src['service_type']} "null"
156 # TODO: Add verification for MCAST
157 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
158 ... Wait Until Keyword Succeeds 60s 2s
159 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
160 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
161 END
162 # Wait for the OLT to come back up
163 FOR ${I} IN RANGE 0 ${olt_count}
164 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
165 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
166 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
167 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
168 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
169 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
170 ... Check Remote System Reachability True ${olt_ssh_ip}
171 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
172 ... Validate OLT Device ENABLED ACTIVE
173 ... REACHABLE ${olt_serial_number}
174 END
175 Run Keyword If ${has_dataplane} Clean Up Linux
176 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
177
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000178Verify ONU Soft Reboot for TT
179 [Documentation] Test the ONU Soft Reboot functionality.
180 [Tags] functionalTT OnuSoftRebootTT
181 [Setup] Start Logging SoftRebootOnu_TT
182 [Teardown] Run Keywords Collect Logs
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000183 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700184 ... AND Collect Logs
185 ... AND Stop Logging SoftRebootOnu_TT
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000186 # Add OLT device
187 Setup
188 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
189 Run Keyword If ${has_dataplane} Clean Up Linux
190 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
191 FOR ${I} IN RANGE 0 ${num_all_onus}
192 ${src}= Set Variable ${hosts.src[${I}]}
193 ${dst}= Set Variable ${hosts.dst[${I}]}
194 ${service_type}= Get Variable Value ${src['service_type']} "null"
195 ${of_id}= Get ofID From OLT List ${src['olt']}
196 ${onu_device_id}= Get Device ID From SN ${src['onu']}
197 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
198 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
199 Reboot Device ${onu_device_id}
200 # TODO: Add verification for MCAST
201 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
202 ... Wait Until Keyword Succeeds ${timeout} 2s
203 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
204 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
205 # Remove Subscriber Access (To replicate TT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000206 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000207 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
208 # Check ONU port is Enabled in ONOS
209 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000210 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000211 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000212 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass8a8d81c2021-04-22 15:33:42 +0000213 ... volt-add-subscriber-access ${of_id} ${onu_port}
214 # Verify ONU state in voltha
215 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
216 ... ENABLED ACTIVE REACHABLE
217 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
218 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
219 ... Run Keyword And Continue On Failure Validate DHCP and Ping True
220 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
221 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
222 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
223 ... ${dst['container_name']}
224 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
225 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
226 ... ${dst} ${suppressaddsubscriber}
227 END
228 Run Keyword If ${has_dataplane} Clean Up Linux
229 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
230
Hardik Windlass656661a2021-04-23 03:46:31 +0000231Verify OLT Soft Reboot for TT
232 [Documentation] Test the OLT Soft Reboot functionality.
233 [Tags] functionalTT OltSoftRebootTT
234 [Setup] Start Logging SoftRebootOlt_TT
235 [Teardown] Run Keywords Collect Logs
Hardik Windlass656661a2021-04-23 03:46:31 +0000236 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700237 ... AND Collect Logs
238 ... AND Stop Logging SoftRebootOlt_TT
Hardik Windlass656661a2021-04-23 03:46:31 +0000239 # Add OLT device
240 Setup
241 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
242 Run Keyword If ${has_dataplane} Clean Up Linux
243 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
244 # Reboot the OLT from the OLT CLI
245 FOR ${I} IN RANGE 0 ${olt_count}
246 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
247 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
248 # Reboot the OLT using "voltctl device reboot" command
249 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200250 # Wait for the OLT to actually go down
251 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
252 ... ${olt_serial_number}
Hardik Windlass656661a2021-04-23 03:46:31 +0000253 END
254 FOR ${I} IN RANGE 0 ${num_all_onus}
255 ${src}= Set Variable ${hosts.src[${I}]}
256 ${dst}= Set Variable ${hosts.dst[${I}]}
257 ${service_type}= Get Variable Value ${src['service_type']} "null"
258 # TODO: Add verification for MCAST
259 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
260 ... Wait Until Keyword Succeeds ${timeout} 2s
261 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
262 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
263 END
264 # Wait for the OLT to come back up
265 FOR ${I} IN RANGE 0 ${olt_count}
266 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
267 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
268 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
269 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
270 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
271 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
272 ... Check Remote System Reachability True ${olt_ssh_ip}
273 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
274 ... Validate OLT Device ENABLED ACTIVE
275 ... REACHABLE ${olt_serial_number}
276 END
277 Run Keyword If ${has_dataplane} Clean Up Linux
278 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
279
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000280Verify restart openolt-adapter container before subscriber provisioning for TT
281 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
282 [Tags] functionalTT Restart-OpenOlt-Before-Subscription-TT
283 [Setup] Start Logging Restart-OpenOlt-Before-Subscription-TT
284 [Teardown] Run Keywords Collect Logs
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700285 ... AND Stop Logging Restart-OpenOlt-Before-Subscription-TT
Emrehan UZUN2884ed52021-05-04 15:36:31 +0000286 # Add OLT device
287 Setup
288 Run Keyword If ${has_dataplane} Clean Up Linux
289 FOR ${I} IN RANGE 0 ${olt_count}
290 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
291 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
292 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
293 ... ${olt_serial_number}
294 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
295 ... Get NNI Port in ONOS ${of_id}
296 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
297 ... Verify Default Downstream Flows are added in ONOS for OLT TT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
298 ... ${nni_port}
299 END
300 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
301 Log ${podStatusOutput}
302 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
303 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
304 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
305 Sleep 5s
306 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
307 ... app ${podName} Running
308 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
309 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
310 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
311 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
312 Log to console Pod ${podName} restarted and sanity checks passed successfully
313
Hardik Windlassd3f63892021-03-25 08:28:02 +0000314Verify restart ofagent container after subscriber is provisioned for TT
315 [Documentation] Restart ofagent container after VOLTHA is operational.
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000316 [Tags] functionalTT ofagentRestart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000317 [Setup] Start Logging ofagentRestart-TT
318 [Teardown] Run Keywords Collect Logs
319 ... AND Stop Logging ofagentRestart-TT
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700320 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000321 # set timeout value
322 ${waitforRestart} Set Variable 120s
323 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
324 Log ${podStatusOutput}
325 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
326 ${podName} Set Variable ofagent
327 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
328 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
329 ... app ${podName} Running
330 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
331 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000332 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT ${suppressaddsubscriber}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000333 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
334 Log ${podStatusOutput}
335 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
336 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
337 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700338 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlassd3f63892021-03-25 08:28:02 +0000339 Sleep 30s
340 FOR ${I} IN RANGE 0 ${num_all_onus}
341 ${src}= Set Variable ${hosts.src[${I}]}
342 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000343 ${service_type}= Get Variable Value ${src['service_type']} "null"
Hardik Windlassd3f63892021-03-25 08:28:02 +0000344 ${of_id}= Get ofID From OLT List ${src['olt']}
345 ${onu_device_id}= Get Device ID From SN ${src['onu']}
346 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
347 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
348 # Verify ONU state in voltha
349 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
350 ... ENABLED ACTIVE REACHABLE
351 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
352 # Check ONU port is Disabled in ONOS
353 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000354 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlassd3f63892021-03-25 08:28:02 +0000355 # Verify Ping
Hardik Windlassa9ec90d2021-05-27 11:33:22 +0000356 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
357 ... Run Keyword And Continue On Failure Check Ping True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000358 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
359 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
360 END
361 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700362 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassd3f63892021-03-25 08:28:02 +0000363 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
364 ... Running
365 Run Keyword If ${has_dataplane} Clean Up Linux
366 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT ${suppressaddsubscriber}
367 Log to console Pod ${podName} restarted and sanity checks passed successfully
368
369Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for TT
370 [Documentation] Deploys an device instance. After that rw-core deployment is scaled to 0 instances to
371 ... simulate a POD crash. The test then scales the rw-core back to a single instance
372 ... and configures ONOS for access. The test succeeds if the device is able to
373 ... complete the DHCP sequence.
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000374 [Tags] functionalTT rwcore-restart-TT
Hardik Windlassd3f63892021-03-25 08:28:02 +0000375 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-TT
376 ... AND Clear All Devices Then Create New Device
377 [Teardown] Run Keywords Collect Logs
378 ... AND Stop Logging RwCoreFailAndRestart-TT
379 Run Keyword If ${has_dataplane} Clean Up Linux
380 FOR ${I} IN RANGE 0 ${olt_count}
381 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
382 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
383 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
384 ... ${olt_serial_number}
385 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
386 ... Get NNI Port in ONOS ${of_id}
387 END
388 FOR ${I} IN RANGE 0 ${num_all_onus}
389 ${src}= Set Variable ${hosts.src[${I}]}
390 ${dst}= Set Variable ${hosts.dst[${I}]}
391 ${of_id}= Get ofID From OLT List ${src['olt']}
392 ${onu_device_id}= Get Device ID From SN ${src['onu']}
393 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
394 ... ${of_id}
395 # Bring up the device and verify it authenticates
396 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000397 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlassd3f63892021-03-25 08:28:02 +0000398 END
399
400 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
401 Scale K8s Deployment voltha voltha-voltha-rw-core 0
402 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
403 # Ensure the ofagent POD goes "not-ready" as expected
404 Wait Until keyword Succeeds ${timeout} 2s
405 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
406 # Scale up the core deployment and make sure both it and the ofagent deployment are back
407 Scale K8s Deployment voltha voltha-voltha-rw-core 1
408 Wait Until Keyword Succeeds ${timeout} 2s
409 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
410 Wait Until Keyword Succeeds ${timeout} 2s
411 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
412 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
413 # so restart the port forwarding for the API service
414 Restart VOLTHA Port Forward voltha-api
415 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
416 # represents system connectivity being restored
417 FOR ${I} IN RANGE 0 ${olt_count}
418 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
419 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
420 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
421 ... ${olt_serial_number}
422 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
423 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
424 END
Hardik Windlass3e3c9942021-04-09 11:37:15 +0000425 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Tests TT
TorstenThieme136ebf72021-02-26 14:27:07 +0000426
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000427Verify restart openonu-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000428 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
429 ... Run the ping continuously in background during container restart,
430 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000431 ... Also, verify that the voltha control plane functionality is not affected.
432 [Tags] functionalTT Restart-OpenOnu-Ping-TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000433 [Setup] Start Logging Restart-OpenOnu-Ping-TT
434 [Teardown] Run Keywords Collect Logs
435 ... AND Stop Logging Restart-OpenOnu-Ping-TT
436 Clear All Devices Then Create New Device
437 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
438 Run Keyword If ${has_dataplane} Clean Up Linux
439 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT
440 FOR ${I} IN RANGE 0 ${num_all_onus}
441 ${src}= Set Variable ${hosts.src[${I}]}
442 ${dst}= Set Variable ${hosts.dst[${I}]}
443 ${service_type}= Get Variable Value ${src['service_type']} "null"
444 Continue For Loop If '${service_type}' == 'mcast'
445 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
446 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
447 ... Wait Until Keyword Succeeds 60s 2s
448 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
449 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
450 END
451 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
452 Log ${podStatusOutput}
453 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
454 ${podName} Set Variable adapter-open-onu
455 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
456 Sleep 5s
457 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
458 ... app ${podName} Running
459 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
460 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
461 Log ${podStatusOutput}
462 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
463 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
464 FOR ${I} IN RANGE 0 ${num_all_onus}
465 ${src}= Set Variable ${hosts.src[${I}]}
466 ${dst}= Set Variable ${hosts.dst[${I}]}
467 ${service_type}= Get Variable Value ${src['service_type']} "null"
468 Continue For Loop If '${service_type}' == 'mcast'
469 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
470 ... Wait Until Keyword Succeeds 60s 2s
471 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
472 ... ${src['container_type']} ${src['container_name']}
473 END
474 FOR ${I} IN RANGE 0 ${num_all_onus}
475 ${src}= Set Variable ${hosts.src[${I}]}
476 ${dst}= Set Variable ${hosts.dst[${I}]}
477 ${service_type}= Get Variable Value ${src['service_type']} "null"
478 Continue For Loop If '${service_type}' == 'mcast'
479 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
480 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
481 ... Wait Until Keyword Succeeds 60s 2s
482 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
483 ... ${src['container_type']} ${src['container_name']}
484 Check Ping Result True ${ping_output}
485 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000486 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
487 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000488
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000489Verify restart openolt-adapter container for TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000490 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
491 ... Run the ping continuously in background during container restart,
492 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000493 ... Also, verify that the voltha control plane functionality is not affected.
494 [Tags] functionalTT Restart-OpenOlt-Ping-TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000495 [Setup] Start Logging Restart-OpenOlt-Ping-TT
496 [Teardown] Run Keywords Collect Logs
497 ... AND Stop Logging Restart-OpenOlt-Ping-TT
498 Clear All Devices Then Create New Device
499 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
500 Run Keyword If ${has_dataplane} Clean Up Linux
501 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT
502 FOR ${I} IN RANGE 0 ${num_all_onus}
503 ${src}= Set Variable ${hosts.src[${I}]}
504 ${dst}= Set Variable ${hosts.dst[${I}]}
505 ${service_type}= Get Variable Value ${src['service_type']} "null"
506 Continue For Loop If '${service_type}' == 'mcast'
507 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
508 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
509 ... Wait Until Keyword Succeeds 60s 2s
510 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
511 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
512 END
513 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
514 Log ${podStatusOutput}
515 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
516 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
517 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
518 Sleep 5s
519 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
520 ... app ${podName} Running
521 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
522 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
523 Log ${podStatusOutput}
524 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
525 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
526 FOR ${I} IN RANGE 0 ${num_all_onus}
527 ${src}= Set Variable ${hosts.src[${I}]}
528 ${dst}= Set Variable ${hosts.dst[${I}]}
529 ${service_type}= Get Variable Value ${src['service_type']} "null"
530 Continue For Loop If '${service_type}' == 'mcast'
531 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
532 ... Wait Until Keyword Succeeds 60s 2s
533 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
534 ... ${src['container_type']} ${src['container_name']}
535 END
536 FOR ${I} IN RANGE 0 ${num_all_onus}
537 ${src}= Set Variable ${hosts.src[${I}]}
538 ${dst}= Set Variable ${hosts.dst[${I}]}
539 ${service_type}= Get Variable Value ${src['service_type']} "null"
540 Continue For Loop If '${service_type}' == 'mcast'
541 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
542 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
543 ... Wait Until Keyword Succeeds 60s 2s
544 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
545 ... ${src['container_type']} ${src['container_name']}
546 Check Ping Result True ${ping_output}
547 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000548 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
549 Verify Control Plane After Pod Restart TT
550
551Verify restart rw-core container for TT
552 [Documentation] Restart rw-core container after VOLTHA is operational.
553 ... Run the ping continuously in background during container restart,
554 ... and verify that there should be no affect on the dataplane.
555 ... Also, verify that the voltha control plane functionality is not affected.
556 [Tags] functionalTT Restart-RwCore-Ping-TT
557 [Setup] Start Logging Restart-RwCore-Ping-TT
558 [Teardown] Run Keywords Collect Logs
559 ... AND Stop Logging Restart-RwCore-Ping-TT
560 Clear All Devices Then Create New Device
561 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
562 Run Keyword If ${has_dataplane} Clean Up Linux
563 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test TT
564 FOR ${I} IN RANGE 0 ${num_all_onus}
565 ${src}= Set Variable ${hosts.src[${I}]}
566 ${dst}= Set Variable ${hosts.dst[${I}]}
567 ${service_type}= Get Variable Value ${src['service_type']} "null"
568 Continue For Loop If '${service_type}' == 'mcast'
569 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
570 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
571 ... Wait Until Keyword Succeeds 60s 2s
572 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
573 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
574 END
575 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
576 Log ${podStatusOutput}
577 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
578 ${podName} Set Variable rw-core
579 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
580 Sleep 5s
581 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
582 ... app ${podName} Running
583 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
584 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
585 # so restart the port forwarding for the API service
586 Restart VOLTHA Port Forward voltha-api
587 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
588 Log ${podStatusOutput}
589 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
590 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
591 FOR ${I} IN RANGE 0 ${num_all_onus}
592 ${src}= Set Variable ${hosts.src[${I}]}
593 ${dst}= Set Variable ${hosts.dst[${I}]}
594 ${service_type}= Get Variable Value ${src['service_type']} "null"
595 Continue For Loop If '${service_type}' == 'mcast'
596 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
597 ... Wait Until Keyword Succeeds 60s 2s
598 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
599 ... ${src['container_type']} ${src['container_name']}
600 END
601 FOR ${I} IN RANGE 0 ${num_all_onus}
602 ${src}= Set Variable ${hosts.src[${I}]}
603 ${dst}= Set Variable ${hosts.dst[${I}]}
604 ${service_type}= Get Variable Value ${src['service_type']} "null"
605 Continue For Loop If '${service_type}' == 'mcast'
606 ${ping_output_file}= Set Variable /tmp/${src['onu']}_${service_type}_ping
607 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
608 ... Wait Until Keyword Succeeds 60s 2s
609 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
610 ... ${src['container_type']} ${src['container_name']}
611 Check Ping Result True ${ping_output}
612 END
613 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
614 Verify Control Plane After Pod Restart TT
Hardik Windlassd50b0202021-09-02 14:45:09 +0000615
TorstenThieme136ebf72021-02-26 14:27:07 +0000616*** Keywords ***
617Setup Suite
618 [Documentation] Set up the test suite
619 Common Test Suite Setup
620 ${switch_type}= Get Variable Value ${web_power_switch.type}
621 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
622
TorstenThieme136ebf72021-02-26 14:27:07 +0000623Teardown Suite
624 [Documentation] Tear down steps for the suite
625 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass04896f62021-03-05 16:15:11 +0530626 Run Keyword If ${teardown_device} Delete All Devices and Verify
TorstenThieme4e2168e2021-06-22 14:01:47 +0000627 Close All ONOS SSH Connections
Hardik Windlassd3f63892021-03-25 08:28:02 +0000628
629Clear All Devices Then Create New Device
630 [Documentation] Remove any devices from VOLTHA and ONOS & then Create new devices
631 # Remove all devices from voltha and onos
632 Delete All Devices and Verify
633 # Execute normal test Setup Keyword
634 Setup
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000635
636Verify Control Plane After Pod Restart TT
637 [Documentation] Verifies the control plane functionality after the voltha pod restart
638 ... by deleting and re-adding the subscriber
639 Run Keyword If ${has_dataplane} Clean Up Linux
640 FOR ${I} IN RANGE 0 ${num_all_onus}
641 ${src}= Set Variable ${hosts.src[${I}]}
642 ${dst}= Set Variable ${hosts.dst[${I}]}
643 ${service_type}= Get Variable Value ${src['service_type']} "null"
644 ${of_id}= Get ofID From OLT List ${src['olt']}
645 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
646 ... ${of_id} ${src['uni_id']}
647 # Remove Subscriber Access
648 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
649 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
650 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast'
651 ... Wait Until Keyword Succeeds ${timeout} 2s
652 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
653 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
654 # Add Subscriber Access
655 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
656 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
657 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
658 ... Validate Device ENABLED ACTIVE
659 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
660 Run Keyword If ${has_dataplane} and '${service_type}' != 'mcast' Run Keyword And Continue On Failure
661 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT one ONU ${src}
662 ... ${dst} ${suppressaddsubscriber}
663 ... ELSE IF ${has_dataplane} and '${service_type}' == 'mcast' Run Keyword And Continue On Failure
664 ... Wait Until Keyword Succeeds ${timeout} 2s Sanity Test TT MCAST one ONU ${src}
665 ... ${dst} ${suppressaddsubscriber}
666 END