blob: 9417b8a258c1d15683cec6e812b75cef5c245675 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
Hardik Windlassb9bdd502020-04-17 14:46:21 +053047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
54${teardown_device} False
55${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Hardik Windlass43531332021-05-19 07:07:03 +000060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
Hardik Windlass4ed54502021-05-14 07:34:15 +000064# Flag specific to Soak Jobs
65${SOAK_TEST} False
66
Hardik Windlassb9bdd502020-04-17 14:46:21 +053067*** Test Cases ***
68Verify ONU after Rebooting Physically for DT
69 [Documentation] Test the ONU functionality by physically turning on/off ONU.
70 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
71 ... Test case runs only on the PODs that are configured with PowerSwitch that
72 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
73 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053074 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlass43531332021-05-19 07:07:03 +000075 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070077 ... AND Run Keyword If ${logging} Collect Logs
78 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053079 # Add OLT device
80 Setup
81 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
82 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +000083 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053085 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080086 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053087 ${src}= Set Variable ${hosts.src[${I}]}
88 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080089 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +000090 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass0560f802020-05-06 23:06:53 +053091 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +000092 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +000093 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Girish Gowdraa3a05c62022-02-16 13:54:37 -080094 # If the power switch port is not specified, continue
95 Continue For Loop If '${src["power_switch_port"]}' == '${None}'
Hardik Windlass7c801e62020-05-04 19:33:21 +053096 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053097 Disable Switch Outlet ${src['power_switch_port']}
98 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +000099 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530100 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
101 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000103 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530104 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Girish Gowdrae0c711e2021-07-20 14:30:17 -0700105 # Allow the remove subscriber command to clean up all the flows, schedulers and queues before deleting the device
Girish Gowdrab6830612021-10-18 17:11:23 -0700106 Sleep 30s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530107 # Delete ONU Device (To replicate DT workflow)
108 Delete Device ${onu_device_id}
109 Sleep 5s
110 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530111 Enable Switch Outlet ${src['power_switch_port']}
112 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000113 Wait Until Keyword Succeeds 120s 5s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000114 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000115 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000116 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530117 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530118 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000119 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530120 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530121 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530122 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000123 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530124 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
125 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530126 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530127 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
128 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530129 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530130 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
131 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530132 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800133 # Verify flows for all OLTs
134 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530135
136Verify OLT after Rebooting Physically for DT
137 [Documentation] Test the physical reboot of the OLT
138 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
139 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
140 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530141 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000142 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800143 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700144 ... AND Run Keyword If ${logging} Collect Logs
145 ... AND Stop Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530146 # Add OLT device
147 Setup
148 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
149 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000150 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530151 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800152 FOR ${I} IN RANGE 0 ${olt_count}
153 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
154 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
155 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
156 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
157 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
158 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100159 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800160 END
161 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530162 ${src}= Set Variable ${hosts.src[${I}]}
163 ${dst}= Set Variable ${hosts.dst[${I}]}
164 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000165 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530166 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
167 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
168 END
169 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800170 FOR ${I} IN RANGE 0 ${olt_count}
171 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
172 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
173 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
174 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
175 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
176 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
177 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000178 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800179 ... Validate OLT Device ENABLED ACTIVE
180 ... REACHABLE ${olt_serial_number}
181 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530182 # Waiting extra time for the ONUs to come up
183 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530184 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000185 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530186
TorstenThiemeed3332f2021-02-02 10:30:48 +0000187Verify restart openonu-adapter container after subscriber provisioning for DT
188 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
189 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000190 [Tags] Restart-OpenOnu-Dt soak
TorstenThiemeed3332f2021-02-02 10:30:48 +0000191 [Setup] Start Logging Restart-OpenOnu-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000192 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
TorstenThiemeed3332f2021-02-02 10:30:48 +0000193 ... AND Stop Logging Restart-OpenOnu-Dt
TorstenThiemeed3332f2021-02-02 10:30:48 +0000194 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000195 Run Keyword If '${SOAK_TEST}'=='False' Setup
TorstenThiemeed3332f2021-02-02 10:30:48 +0000196 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
197 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000198 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000199 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
200 Log ${podStatusOutput}
201 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
202 ${podName} Set Variable adapter-open-onu
203 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000204 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
205 ... app ${podName} Running
206 # Wait for 1 min after openonu adapter is restarted
207 Sleep 60s
208 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000209 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000210 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
211 Log ${podStatusOutput}
212 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
213 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
214 Log to console Pod ${podName} restarted and sanity checks passed successfully
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000215 Run Keyword If '${SOAK_TEST}'=='False' Delete All Devices and Verify
TorstenThiemeed3332f2021-02-02 10:30:48 +0000216
Hardik Windlass9df139e2020-04-24 14:54:54 +0530217Verify restart openolt-adapter container after subscriber provisioning for DT
218 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
219 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000220 [Tags] Restart-OpenOlt-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530221 [Setup] Start Logging Restart-OpenOlt-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000222 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530223 ... AND Stop Logging Restart-OpenOlt-Dt
224 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000225 Run Keyword If '${SOAK_TEST}'=='False' setup
Hardik Windlass9df139e2020-04-24 14:54:54 +0530226 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
227 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000228 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530229 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
230 Log ${podStatusOutput}
231 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100232 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530233 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530234 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
235 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000236 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530237 Sleep 60s
238 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000239 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530240 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
241 Log ${podStatusOutput}
242 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
243 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
244 Log to console Pod ${podName} restarted and sanity checks passed successfully
245
246Verify openolt adapter restart before subscriber provisioning for DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000247 [Documentation] Restart openolt-adapter container before adding the subscriber.
248 [Tags] functionalDt olt-adapter-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530249 [Setup] Start Logging OltAdapterRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000250 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530251 ... AND Stop Logging OltAdapterRestart-Dt
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000252 # Add OLT device
253 Clear All Devices Then Create New Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530254 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass9df139e2020-04-24 14:54:54 +0530255 Set Global Variable ${of_id}
256
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800257 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530258 ${src}= Set Variable ${hosts.src[${I}]}
259 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800260 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530261 ${onu_device_id}= Get Device ID From SN ${src['onu']}
262 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000263 ... ${of_id} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000264 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000265 ... Validate Device ENABLED ACTIVE REACHABLE
266 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530267 END
268 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100269 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
270 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
271 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530272 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100273 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530274 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100275 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530276
277 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800278 FOR ${I} IN RANGE 0 ${olt_count}
279 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
280 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
281 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
282 ... ${olt_serial_number}
283 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
284 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
285 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530286
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800287 FOR ${I} IN RANGE 0 ${num_all_onus}
288 ${src}= Set Variable ${hosts.src[${I}]}
289 ${dst}= Set Variable ${hosts.dst[${I}]}
290 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000291 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800292 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000293 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530294 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000295 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530296 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
297 # Verify subscriber access flows are added for the ONU port
298 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
299 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
300 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000301 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000302 ... ENABLED ACTIVE REACHABLE
303 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass9df139e2020-04-24 14:54:54 +0530304 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
305 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
306 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
307 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
308 ... ${dst['container_name']}
309 END
310
311Verify restart ofagent container after subscriber is provisioned for DT
312 [Documentation] Restart ofagent container after VOLTHA is operational.
313 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000314 [Tags] functionalDt ofagentRestart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530315 [Setup] Start Logging ofagentRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000316 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530317 ... AND Stop Logging ofagentRestart-Dt
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700318 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530319 # set timeout value
320 ${waitforRestart} Set Variable 120s
321 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
322 Log ${podStatusOutput}
323 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
324 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530325 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530326 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
327 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000328 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530329 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
330 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000331 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530332 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
333 Log ${podStatusOutput}
334 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
335 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
336 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700337 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530338 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800339 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530340 ${src}= Set Variable ${hosts.src[${I}]}
341 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800342 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000343 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530344 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000345 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000346 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530347 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000348 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530349 ... ENABLED ACTIVE REACHABLE
350 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
351 # Check ONU port is Disabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000352 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000353 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530354 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000355 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530356 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
357 ... ${onu_port} ${nni_port} ${src['s_tag']}
358 # Verify Ping
359 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
360 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
361 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530362 END
363 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700364 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530365 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
366 ... Running
367 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
368 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000369 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530370 Log to console Pod ${podName} restarted and sanity checks passed successfully
371
372Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
373 [Documentation] Deploys an device instance and waits for it to authenticate. After
374 ... authentication is successful the rw-core deployment is scaled to 0 instances to
375 ... simulate a POD crash. The test then scales the rw-core back to a single instance
376 ... and configures ONOS for access. The test succeeds if the device is able to
377 ... complete the DHCP sequence.
Hardik Windlass43531332021-05-19 07:07:03 +0000378 [Tags] functionalDt rwcore-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530379 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
380 ... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000381 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530382 ... AND Stop Logging RwCoreFailAndRestart-Dt
383 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530384 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800385 FOR ${I} IN RANGE 0 ${olt_count}
386 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
387 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
388 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
389 ... ${olt_serial_number}
TorstenThiemedda37652021-11-11 16:22:05 +0000390 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800391 END
392 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530393 ${src}= Set Variable ${hosts.src[${I}]}
394 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800395 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530396 ${onu_device_id}= Get Device ID From SN ${src['onu']}
397 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000398 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530399 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700400 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000401 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530402 END
403
404 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530405 Scale K8s Deployment voltha voltha-voltha-rw-core 0
406 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530407 # Ensure the ofagent POD goes "not-ready" as expected
408 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530409 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530410 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530411 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530412 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530413 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530414 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530415 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530416 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
417 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700418 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530419 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
420 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800421 FOR ${I} IN RANGE 0 ${olt_count}
422 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
423 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
424 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
425 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700426 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800427 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
428 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530429
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800430 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700431 ${src}= Set Variable ${hosts.src[${I}]}
432 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800433 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000434 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Andy Bavier52094622020-05-12 15:54:24 -0700435 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000436 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530437 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000438 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530439 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
440 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000441 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530442 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
443 ... ${onu_port} ${nni_port} ${src['s_tag']}
444 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
445 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
446 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
447 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
448 ... ${dst['container_name']}
449 END
450
Hemadf003682020-04-28 21:22:22 +0530451Verify OLT Soft Reboot for DT
452 [Documentation] Test soft reboot of the OLT using voltctl command
Hardik Windlass20a503d2021-06-21 11:58:08 +0000453 [Tags] VOL-2818 OLTSoftRebootDt functionalDt
Hemadf003682020-04-28 21:22:22 +0530454 [Setup] Start Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000455 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hemadf003682020-04-28 21:22:22 +0530456 ... AND Stop Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000457 FOR ${I} IN RANGE 0 ${olt_count}
458 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
459 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
460 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
461 ... Validate OLT Device ENABLED ACTIVE
462 ... REACHABLE ${olt_serial_number}
463 # Reboot the OLT using "voltctl device reboot" command
464 Reboot Device ${olt_device_id}
465 # Wait for the OLT to actually go down
466 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
467 ... ${olt_serial_number}
468 END
Hemadf003682020-04-28 21:22:22 +0530469 #Verify that ping fails
Hardik Windlass20a503d2021-06-21 11:58:08 +0000470 FOR ${I} IN RANGE 0 ${num_all_onus}
Hemadf003682020-04-28 21:22:22 +0530471 ${src}= Set Variable ${hosts.src[${I}]}
472 ${dst}= Set Variable ${hosts.dst[${I}]}
473 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000474 ... Wait Until Keyword Succeeds ${timeout} 2s
Hemadf003682020-04-28 21:22:22 +0530475 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
476 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
477 END
Hardik Windlass20a503d2021-06-21 11:58:08 +0000478 # Check OLT states
479 FOR ${I} IN RANGE 0 ${olt_count}
480 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
481 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
482 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
483 # Wait for the OLT to come back up
484 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
485 ... Check Remote System Reachability True ${olt_ssh_ip}
486 # Check OLT states
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000487 Wait Until Keyword Succeeds 360s 5s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000488 ... Validate OLT Device ENABLED ACTIVE
489 ... REACHABLE ${olt_serial_number}
490 END
Hemadf003682020-04-28 21:22:22 +0530491 # Waiting extra time for the ONUs to come up
492 Sleep 60s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000493 #Check after reboot that ONUs are active, DHCP/pingable
Hemadf003682020-04-28 21:22:22 +0530494 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000495 Perform Sanity Test DT
Hemadf003682020-04-28 21:22:22 +0530496
Hema8c1a0322020-05-26 16:02:32 +0530497Verify ONU Soft Reboot for DT
498 [Documentation] Test soft reboot of the ONU using voltctl command
499 [Tags] VOL-2820 ONUSoftRebootDt notready
500 [Setup] Start Logging ONUSoftRebootDt
501 #... AND Setup
502 [Teardown] Run Keywords Collect Logs
503 ... AND Stop Logging ONUSoftRebootDt
504 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530505 #Reboot the ONU and verify that ping fails
506 FOR ${I} IN RANGE 0 ${num_onus}
507 ${src}= Set Variable ${hosts.src[${I}]}
508 ${dst}= Set Variable ${hosts.dst[${I}]}
TorstenThiemedda37652021-11-11 16:22:05 +0000509 ${of_id}= Get ofID From OLT List ${src['olt']}
510 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hema8c1a0322020-05-26 16:02:32 +0530511 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000512 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000513 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hema8c1a0322020-05-26 16:02:32 +0530514 Reboot Device ${onu_device_id}
515 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000516 ... Wait Until Keyword Succeeds ${timeout} 2s
Hema8c1a0322020-05-26 16:02:32 +0530517 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
518 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlassb75cf1b2021-01-18 06:59:09 +0000519 Verify ping is successful except for given device ${num_onus} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530520 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000521 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hema8c1a0322020-05-26 16:02:32 +0530522 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
523 # Delete ONU Device (To replicate DT workflow)
524 Delete Device ${onu_device_id}
525 Sleep 40s
526 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000527 Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000528 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000529 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000530 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hema8c1a0322020-05-26 16:02:32 +0530531 ... volt-add-subscriber-access ${of_id} ${onu_port}
532 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700533 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530534 ... ENABLED ACTIVE REACHABLE
535 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
536 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000537 Wait Until Keyword Succeeds ${timeout} 5s
Hema8c1a0322020-05-26 16:02:32 +0530538 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
539 ... ${onu_port} ${nni_port} ${src['s_tag']}
540 #Run Keyword If ${has_dataplane} Clean Up Linux
541 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
542 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
543 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
544 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
545 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530546 END
547 # Verify ONOS Flows
548 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
549 ${onos_flows_count}= Evaluate 4 * ${num_onus}
550 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
551 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
552 ... ${of_id} ${onos_flows_count}
553 # Verify VOLTHA Flows
554 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
555 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
556 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
557 ${onu_flows}= Set Variable 2
558 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
559 ${List_ONU_Serial} Create List
560 Set Suite Variable ${List_ONU_Serial}
561 Build ONU SN List ${List_ONU_Serial}
562 Log ${List_ONU_Serial}
563 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
564 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530565
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000566Verify restart openonu-adapter container for DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000567 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
568 ... Run the ping continuously in background during container restart,
569 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000570 ... Also, verify that the voltha control plane functionality is not affected.
571 [Tags] functionalDt RestartOpenOnuPingDt
Hardik Windlassd19af862021-08-12 11:40:20 +0000572 [Setup] Start Logging RestartOpenOnuPingDt
573 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
574 ... AND Stop Logging RestartOpenOnuPingDt
575 Clear All Devices Then Create New Device
576 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
577 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000578 Perform Sanity Test DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000579 FOR ${I} IN RANGE 0 ${num_all_onus}
580 ${src}= Set Variable ${hosts.src[${I}]}
581 ${dst}= Set Variable ${hosts.dst[${I}]}
582 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
583 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000584 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000585 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
586 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
587 END
588 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
589 Log ${podStatusOutput}
590 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
591 ${podName} Set Variable adapter-open-onu
592 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
593 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
594 ... app ${podName} Running
595 # Wait for 1 min after openonu adapter is restarted
596 Sleep 60s
597 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
598 Log ${podStatusOutput}
599 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
600 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
601 FOR ${I} IN RANGE 0 ${num_all_onus}
602 ${src}= Set Variable ${hosts.src[${I}]}
603 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassd19af862021-08-12 11:40:20 +0000604 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000605 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000606 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
607 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000608 END
609 FOR ${I} IN RANGE 0 ${num_all_onus}
610 ${src}= Set Variable ${hosts.src[${I}]}
611 ${dst}= Set Variable ${hosts.dst[${I}]}
612 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassd19af862021-08-12 11:40:20 +0000613 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000614 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000615 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
616 ... ${src['container_type']} ${src['container_name']}
617 Check Ping Result True ${ping_output}
618 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000619 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
620 Verify Control Plane After Pod Restart DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000621
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000622Verify restart openolt-adapter container for DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000623 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
624 ... Run the ping continuously in background during container restart,
625 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000626 ... Also, verify that the voltha control plane functionality is not affected.
627 [Tags] functionalDt RestartOpenOltPingDt
Hardik Windlassa04efa52021-08-13 08:17:24 +0000628 [Setup] Start Logging RestartOpenOltPingDt
629 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
630 ... AND Stop Logging RestartOpenOltPingDt
631 Clear All Devices Then Create New Device
632 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
633 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000634 Perform Sanity Test DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000635 FOR ${I} IN RANGE 0 ${num_all_onus}
636 ${src}= Set Variable ${hosts.src[${I}]}
637 ${dst}= Set Variable ${hosts.dst[${I}]}
638 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
639 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000640 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000641 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
642 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
643 END
644 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
645 Log ${podStatusOutput}
646 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
647 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
648 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
649 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
650 ... app ${podName} Running
651 # Wait for 1 min after openolt adapter is restarted
652 Sleep 60s
653 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
654 Log ${podStatusOutput}
655 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
656 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
657 FOR ${I} IN RANGE 0 ${num_all_onus}
658 ${src}= Set Variable ${hosts.src[${I}]}
659 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa04efa52021-08-13 08:17:24 +0000660 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000661 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000662 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
663 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000664 END
665 FOR ${I} IN RANGE 0 ${num_all_onus}
666 ${src}= Set Variable ${hosts.src[${I}]}
667 ${dst}= Set Variable ${hosts.dst[${I}]}
668 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassa04efa52021-08-13 08:17:24 +0000669 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000670 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000671 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
672 ... ${src['container_type']} ${src['container_name']}
673 Check Ping Result True ${ping_output}
674 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000675 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
676 Verify Control Plane After Pod Restart DT
677
678Verify restart rw-core container for DT
679 [Documentation] Restart rw-core container after VOLTHA is operational.
680 ... Run the ping continuously in background during container restart,
681 ... and verify that there should be no affect on the dataplane.
682 ... Also, verify that the voltha control plane functionality is not affected.
683 [Tags] functionalDt RestartRwCorePingDt
684 [Setup] Start Logging RestartRwCorePingDt
685 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
686 ... AND Stop Logging RestartRwCorePingDt
687 Clear All Devices Then Create New Device
688 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
689 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000690 Perform Sanity Test DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000691 FOR ${I} IN RANGE 0 ${num_all_onus}
692 ${src}= Set Variable ${hosts.src[${I}]}
693 ${dst}= Set Variable ${hosts.dst[${I}]}
694 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
695 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000696 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000697 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
698 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
699 END
700 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
701 Log ${podStatusOutput}
702 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
703 ${podName} Set Variable rw-core
704 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
705 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
706 ... app ${podName} Running
707 # Wait for 1 min after rw-core is restarted
708 Sleep 60s
709 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
710 # so restart the port forwarding for the API service
711 Restart VOLTHA Port Forward voltha-api
712 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
713 Log ${podStatusOutput}
714 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
715 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
716 FOR ${I} IN RANGE 0 ${num_all_onus}
717 ${src}= Set Variable ${hosts.src[${I}]}
718 ${dst}= Set Variable ${hosts.dst[${I}]}
719 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000720 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000721 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
722 ... ${src['container_type']} ${src['container_name']}
723 END
724 FOR ${I} IN RANGE 0 ${num_all_onus}
725 ${src}= Set Variable ${hosts.src[${I}]}
726 ${dst}= Set Variable ${hosts.dst[${I}]}
727 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
728 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000729 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000730 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
731 ... ${src['container_type']} ${src['container_name']}
732 Check Ping Result True ${ping_output}
733 END
734 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
735 Verify Control Plane After Pod Restart DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000736
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530737*** Keywords ***
738Setup Suite
739 [Documentation] Set up the test suite
740 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200741 #power_switch.robot needs it to support different vendor's power switch
742 ${switch_type}= Get Variable Value ${web_power_switch.type}
743 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000744 # Run Pre-test Setup for Soak Job
745 # Note: As soak requirement, it expects that the devices under test are already created and enabled
746 Run Keyword If '${SOAK_TEST}'=='True' Setup Soak
747
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530748
749Clear All Devices Then Create New Device
750 [Documentation] Remove any devices from VOLTHA and ONOS
751 # Remove all devices from voltha and nos
752 Delete All Devices and Verify
753 # Execute normal test Setup Keyword
754 Setup
755
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000756Verify Control Plane After Pod Restart DT
757 [Documentation] Verifies the control plane functionality after the voltha pod restart
758 ... by deleting and re-adding the subscriber
759 FOR ${I} IN RANGE 0 ${num_all_onus}
760 ${src}= Set Variable ${hosts.src[${I}]}
761 ${dst}= Set Variable ${hosts.dst[${I}]}
762 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000763 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000764 ${onu_device_id}= Get Device ID From SN ${src['onu']}
765 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000766 ... ${of_id} ${src['uni_id']}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000767 # Remove Subscriber Access
768 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
769 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
770 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000771 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000772 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
773 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
774 # Disable and Re-Enable the ONU (To replicate DT current workflow)
775 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
776 Disable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000777 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000778 ... Validate Device DISABLED UNKNOWN
779 ... REACHABLE ${src['onu']}
780 Enable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000781 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000782 ... Validate Device ENABLED ACTIVE
783 ... REACHABLE ${src['onu']}
784 # Add Subscriber Access
785 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
786 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
787 # Verify subscriber access flows are added for the ONU port
788 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
789 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
790 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000791 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000792 ... Validate Device ENABLED ACTIVE
793 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
Girish Gowdra8fd96962021-12-14 16:20:04 +0530794 # Workaround for issue seen in VOL-4489. Keep this workaround until VOL-4489 is fixed.
795 Run Keyword If ${has_dataplane} Reboot XGSPON ONU ${src['olt']} ${src['onu']} omci-flows-pushed
796 # Workaround ends here for issue seen in VOL-4489.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000797 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000798 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000799 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
800 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
801 END