blob: 54d15aa72281e7b9350c16c6b3b9c794b1c6d6d0 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
Hardik Windlassb9bdd502020-04-17 14:46:21 +053047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
54${teardown_device} False
55${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Hardik Windlass43531332021-05-19 07:07:03 +000060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
Hardik Windlass4ed54502021-05-14 07:34:15 +000064# Flag specific to Soak Jobs
65${SOAK_TEST} False
66
Hardik Windlassb9bdd502020-04-17 14:46:21 +053067*** Test Cases ***
68Verify ONU after Rebooting Physically for DT
69 [Documentation] Test the ONU functionality by physically turning on/off ONU.
70 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
71 ... Test case runs only on the PODs that are configured with PowerSwitch that
72 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
73 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053074 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlass43531332021-05-19 07:07:03 +000075 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070077 ... AND Run Keyword If ${logging} Collect Logs
78 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053079 # Add OLT device
80 Setup
81 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
82 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +000083 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053085 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080086 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053087 ${src}= Set Variable ${hosts.src[${I}]}
88 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080089 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +000090 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass0560f802020-05-06 23:06:53 +053091 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +000092 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +000093 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Girish Gowdraa3a05c62022-02-16 13:54:37 -080094 # If the power switch port is not specified, continue
95 Continue For Loop If '${src["power_switch_port"]}' == '${None}'
Hardik Windlass7c801e62020-05-04 19:33:21 +053096 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053097 Disable Switch Outlet ${src['power_switch_port']}
98 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +000099 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530100 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
101 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000103 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530104 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Girish Gowdrae0c711e2021-07-20 14:30:17 -0700105 # Allow the remove subscriber command to clean up all the flows, schedulers and queues before deleting the device
Girish Gowdrab6830612021-10-18 17:11:23 -0700106 Sleep 30s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530107 # Delete ONU Device (To replicate DT workflow)
108 Delete Device ${onu_device_id}
109 Sleep 5s
110 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530111 Enable Switch Outlet ${src['power_switch_port']}
112 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000113 Wait Until Keyword Succeeds 120s 5s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000114 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000115 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000116 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530117 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530118 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000119 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530120 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530121 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530122 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000123 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530124 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
125 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530126 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530127 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
128 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530129 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530130 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
131 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530132 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800133 # Verify flows for all OLTs
134 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530135
136Verify OLT after Rebooting Physically for DT
137 [Documentation] Test the physical reboot of the OLT
138 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
139 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
140 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530141 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000142 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800143 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700144 ... AND Run Keyword If ${logging} Collect Logs
145 ... AND Stop Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530146 # Add OLT device
147 Setup
148 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
149 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000150 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530151 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800152 FOR ${I} IN RANGE 0 ${olt_count}
153 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
154 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
155 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
156 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
157 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
158 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100159 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800160 END
161 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530162 ${src}= Set Variable ${hosts.src[${I}]}
163 ${dst}= Set Variable ${hosts.dst[${I}]}
164 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000165 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530166 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
167 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
168 END
169 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800170 FOR ${I} IN RANGE 0 ${olt_count}
171 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
172 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
173 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
174 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
175 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
176 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
177 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000178 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800179 ... Validate OLT Device ENABLED ACTIVE
180 ... REACHABLE ${olt_serial_number}
181 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530182 # Waiting extra time for the ONUs to come up
183 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530184 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000185 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530186
TorstenThiemeed3332f2021-02-02 10:30:48 +0000187Verify restart openonu-adapter container after subscriber provisioning for DT
188 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
189 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000190 [Tags] Restart-OpenOnu-Dt soak
TorstenThiemeed3332f2021-02-02 10:30:48 +0000191 [Setup] Start Logging Restart-OpenOnu-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000192 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
TorstenThiemeed3332f2021-02-02 10:30:48 +0000193 ... AND Stop Logging Restart-OpenOnu-Dt
TorstenThiemeed3332f2021-02-02 10:30:48 +0000194 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000195 Run Keyword If '${SOAK_TEST}'=='False' Setup
TorstenThiemeed3332f2021-02-02 10:30:48 +0000196 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
197 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000198 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000199 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
200 Log ${podStatusOutput}
201 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
202 ${podName} Set Variable adapter-open-onu
203 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000204 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
205 ... app ${podName} Running
206 # Wait for 1 min after openonu adapter is restarted
207 Sleep 60s
208 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000209 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000210 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
211 Log ${podStatusOutput}
212 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
213 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
214 Log to console Pod ${podName} restarted and sanity checks passed successfully
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000215 Run Keyword If '${SOAK_TEST}'=='False' Delete All Devices and Verify
TorstenThiemeed3332f2021-02-02 10:30:48 +0000216
Hardik Windlass9df139e2020-04-24 14:54:54 +0530217Verify restart openolt-adapter container after subscriber provisioning for DT
218 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
219 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000220 [Tags] Restart-OpenOlt-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530221 [Setup] Start Logging Restart-OpenOlt-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000222 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530223 ... AND Stop Logging Restart-OpenOlt-Dt
224 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000225 Run Keyword If '${SOAK_TEST}'=='False' setup
Hardik Windlass9df139e2020-04-24 14:54:54 +0530226 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
227 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000228 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530229 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
230 Log ${podStatusOutput}
231 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100232 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530233 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530234 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
235 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000236 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530237 Sleep 60s
238 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000239 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530240 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
241 Log ${podStatusOutput}
242 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
243 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
244 Log to console Pod ${podName} restarted and sanity checks passed successfully
245
246Verify openolt adapter restart before subscriber provisioning for DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000247 [Documentation] Restart openolt-adapter container before adding the subscriber.
248 [Tags] functionalDt olt-adapter-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530249 [Setup] Start Logging OltAdapterRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000250 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530251 ... AND Stop Logging OltAdapterRestart-Dt
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000252 # Add OLT device
253 Clear All Devices Then Create New Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530254 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass9df139e2020-04-24 14:54:54 +0530255 Set Global Variable ${of_id}
256
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800257 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530258 ${src}= Set Variable ${hosts.src[${I}]}
259 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800260 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530261 ${onu_device_id}= Get Device ID From SN ${src['onu']}
262 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000263 ... ${of_id} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000264 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000265 ... Validate Device ENABLED ACTIVE REACHABLE
266 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530267 END
268 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100269 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
270 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
271 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass713c0a62022-04-04 05:33:31 +0000272 Sleep 10s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530273 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100274 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530275 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100276 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass713c0a62022-04-04 05:33:31 +0000277 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530278
279 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800280 FOR ${I} IN RANGE 0 ${olt_count}
281 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
282 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
283 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
284 ... ${olt_serial_number}
285 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
286 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
287 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530288
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800289 FOR ${I} IN RANGE 0 ${num_all_onus}
290 ${src}= Set Variable ${hosts.src[${I}]}
291 ${dst}= Set Variable ${hosts.dst[${I}]}
292 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000293 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800294 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000295 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530296 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000297 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530298 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass713c0a62022-04-04 05:33:31 +0000299 # Verify Meters in ONOS
300 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
301 ... Verify Meters in ONOS Ietf ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530302 # Verify subscriber access flows are added for the ONU port
303 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
304 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
305 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000306 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000307 ... ENABLED ACTIVE REACHABLE
308 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass9df139e2020-04-24 14:54:54 +0530309 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
310 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
311 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
312 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
313 ... ${dst['container_name']}
314 END
315
316Verify restart ofagent container after subscriber is provisioned for DT
317 [Documentation] Restart ofagent container after VOLTHA is operational.
318 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000319 [Tags] functionalDt ofagentRestart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530320 [Setup] Start Logging ofagentRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000321 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530322 ... AND Stop Logging ofagentRestart-Dt
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700323 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530324 ... AND Wait Until Keyword Succeeds ${timeout} 2s
325 ... Validate Pods status By Label ${NAMESPACE} app ofagent Running
326 ... AND Wait Until Keyword Succeeds ${timeout} 3s
327 ... Pods Are Ready By Label ${NAMESPACE} app ofagent
Hardik Windlass9df139e2020-04-24 14:54:54 +0530328 # set timeout value
329 ${waitforRestart} Set Variable 120s
330 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
331 Log ${podStatusOutput}
332 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
333 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530334 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530335 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
336 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000337 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530338 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
339 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000340 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530341 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
342 Log ${podStatusOutput}
343 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
344 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
345 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700346 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530347 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800348 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530349 ${src}= Set Variable ${hosts.src[${I}]}
350 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800351 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000352 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530353 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000354 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000355 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530356 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000357 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530358 ... ENABLED ACTIVE REACHABLE
359 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
360 # Check ONU port is Disabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000361 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000362 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530363 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000364 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530365 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
366 ... ${onu_port} ${nni_port} ${src['s_tag']}
367 # Verify Ping
368 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
369 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
370 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530371 END
372 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700373 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530374 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
375 ... Running
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530376 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530377 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
378 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000379 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530380 Log to console Pod ${podName} restarted and sanity checks passed successfully
381
382Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
383 [Documentation] Deploys an device instance and waits for it to authenticate. After
384 ... authentication is successful the rw-core deployment is scaled to 0 instances to
385 ... simulate a POD crash. The test then scales the rw-core back to a single instance
386 ... and configures ONOS for access. The test succeeds if the device is able to
387 ... complete the DHCP sequence.
Hardik Windlass43531332021-05-19 07:07:03 +0000388 [Tags] functionalDt rwcore-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530389 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
390 ... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000391 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530392 ... AND Stop Logging RwCoreFailAndRestart-Dt
393 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530394 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800395 FOR ${I} IN RANGE 0 ${olt_count}
396 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
397 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
398 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
399 ... ${olt_serial_number}
TorstenThiemedda37652021-11-11 16:22:05 +0000400 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800401 END
402 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530403 ${src}= Set Variable ${hosts.src[${I}]}
404 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800405 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530406 ${onu_device_id}= Get Device ID From SN ${src['onu']}
407 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000408 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530409 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700410 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000411 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530412 END
413
414 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530415 Scale K8s Deployment voltha voltha-voltha-rw-core 0
416 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530417 # Ensure the ofagent POD goes "not-ready" as expected
418 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530419 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530420 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530421 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530422 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530423 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530424 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530425 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530426 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
427 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700428 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530429 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
430 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800431 FOR ${I} IN RANGE 0 ${olt_count}
432 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
433 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
434 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
435 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700436 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800437 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
438 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530439
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800440 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700441 ${src}= Set Variable ${hosts.src[${I}]}
442 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800443 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000444 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Andy Bavier52094622020-05-12 15:54:24 -0700445 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000446 ... ${of_id} ${src['uni_id']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530447 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000448 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530449 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
450 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000451 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530452 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
453 ... ${onu_port} ${nni_port} ${src['s_tag']}
454 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
455 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
456 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
457 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
458 ... ${dst['container_name']}
459 END
460
Hemadf003682020-04-28 21:22:22 +0530461Verify OLT Soft Reboot for DT
462 [Documentation] Test soft reboot of the OLT using voltctl command
Hardik Windlass20a503d2021-06-21 11:58:08 +0000463 [Tags] VOL-2818 OLTSoftRebootDt functionalDt
Hemadf003682020-04-28 21:22:22 +0530464 [Setup] Start Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000465 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hemadf003682020-04-28 21:22:22 +0530466 ... AND Stop Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000467 FOR ${I} IN RANGE 0 ${olt_count}
468 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
469 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
470 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
471 ... Validate OLT Device ENABLED ACTIVE
472 ... REACHABLE ${olt_serial_number}
473 # Reboot the OLT using "voltctl device reboot" command
474 Reboot Device ${olt_device_id}
475 # Wait for the OLT to actually go down
476 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
477 ... ${olt_serial_number}
478 END
Hemadf003682020-04-28 21:22:22 +0530479 #Verify that ping fails
Hardik Windlass20a503d2021-06-21 11:58:08 +0000480 FOR ${I} IN RANGE 0 ${num_all_onus}
Hemadf003682020-04-28 21:22:22 +0530481 ${src}= Set Variable ${hosts.src[${I}]}
482 ${dst}= Set Variable ${hosts.dst[${I}]}
483 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000484 ... Wait Until Keyword Succeeds ${timeout} 2s
Hemadf003682020-04-28 21:22:22 +0530485 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
486 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
487 END
Hardik Windlass20a503d2021-06-21 11:58:08 +0000488 # Check OLT states
489 FOR ${I} IN RANGE 0 ${olt_count}
490 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
491 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
492 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
493 # Wait for the OLT to come back up
494 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
495 ... Check Remote System Reachability True ${olt_ssh_ip}
496 # Check OLT states
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000497 Wait Until Keyword Succeeds 360s 5s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000498 ... Validate OLT Device ENABLED ACTIVE
499 ... REACHABLE ${olt_serial_number}
500 END
Hemadf003682020-04-28 21:22:22 +0530501 # Waiting extra time for the ONUs to come up
502 Sleep 60s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000503 #Check after reboot that ONUs are active, DHCP/pingable
Hemadf003682020-04-28 21:22:22 +0530504 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000505 Perform Sanity Test DT
Hemadf003682020-04-28 21:22:22 +0530506
Hema8c1a0322020-05-26 16:02:32 +0530507Verify ONU Soft Reboot for DT
508 [Documentation] Test soft reboot of the ONU using voltctl command
509 [Tags] VOL-2820 ONUSoftRebootDt notready
510 [Setup] Start Logging ONUSoftRebootDt
511 #... AND Setup
512 [Teardown] Run Keywords Collect Logs
513 ... AND Stop Logging ONUSoftRebootDt
514 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530515 #Reboot the ONU and verify that ping fails
516 FOR ${I} IN RANGE 0 ${num_onus}
517 ${src}= Set Variable ${hosts.src[${I}]}
518 ${dst}= Set Variable ${hosts.dst[${I}]}
TorstenThiemedda37652021-11-11 16:22:05 +0000519 ${of_id}= Get ofID From OLT List ${src['olt']}
520 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hema8c1a0322020-05-26 16:02:32 +0530521 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000522 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000523 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hema8c1a0322020-05-26 16:02:32 +0530524 Reboot Device ${onu_device_id}
525 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000526 ... Wait Until Keyword Succeeds ${timeout} 2s
Hema8c1a0322020-05-26 16:02:32 +0530527 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
528 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlassb75cf1b2021-01-18 06:59:09 +0000529 Verify ping is successful except for given device ${num_onus} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530530 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000531 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hema8c1a0322020-05-26 16:02:32 +0530532 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
533 # Delete ONU Device (To replicate DT workflow)
534 Delete Device ${onu_device_id}
535 Sleep 40s
536 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000537 Wait Until Keyword Succeeds ${timeout} 2s
Emrehan UZUNabef5182021-11-26 10:54:24 +0000538 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000539 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000540 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hema8c1a0322020-05-26 16:02:32 +0530541 ... volt-add-subscriber-access ${of_id} ${onu_port}
542 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700543 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530544 ... ENABLED ACTIVE REACHABLE
545 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
546 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000547 Wait Until Keyword Succeeds ${timeout} 5s
Hema8c1a0322020-05-26 16:02:32 +0530548 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
549 ... ${onu_port} ${nni_port} ${src['s_tag']}
550 #Run Keyword If ${has_dataplane} Clean Up Linux
551 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
552 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
553 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
554 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
555 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530556 END
557 # Verify ONOS Flows
558 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
559 ${onos_flows_count}= Evaluate 4 * ${num_onus}
560 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
561 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
562 ... ${of_id} ${onos_flows_count}
563 # Verify VOLTHA Flows
564 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
565 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
566 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
567 ${onu_flows}= Set Variable 2
568 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
569 ${List_ONU_Serial} Create List
570 Set Suite Variable ${List_ONU_Serial}
571 Build ONU SN List ${List_ONU_Serial}
572 Log ${List_ONU_Serial}
573 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
574 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530575
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000576Verify restart openonu-adapter container for DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000577 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
578 ... Run the ping continuously in background during container restart,
579 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000580 ... Also, verify that the voltha control plane functionality is not affected.
581 [Tags] functionalDt RestartOpenOnuPingDt
Hardik Windlassd19af862021-08-12 11:40:20 +0000582 [Setup] Start Logging RestartOpenOnuPingDt
583 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
584 ... AND Stop Logging RestartOpenOnuPingDt
585 Clear All Devices Then Create New Device
586 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
587 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000588 Perform Sanity Test DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000589 FOR ${I} IN RANGE 0 ${num_all_onus}
590 ${src}= Set Variable ${hosts.src[${I}]}
591 ${dst}= Set Variable ${hosts.dst[${I}]}
592 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
593 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000594 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000595 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
596 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
597 END
598 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
599 Log ${podStatusOutput}
600 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
601 ${podName} Set Variable adapter-open-onu
602 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
603 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
604 ... app ${podName} Running
605 # Wait for 1 min after openonu adapter is restarted
606 Sleep 60s
607 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
608 Log ${podStatusOutput}
609 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
610 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
611 FOR ${I} IN RANGE 0 ${num_all_onus}
612 ${src}= Set Variable ${hosts.src[${I}]}
613 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassd19af862021-08-12 11:40:20 +0000614 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000615 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000616 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
617 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000618 END
619 FOR ${I} IN RANGE 0 ${num_all_onus}
620 ${src}= Set Variable ${hosts.src[${I}]}
621 ${dst}= Set Variable ${hosts.dst[${I}]}
622 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassd19af862021-08-12 11:40:20 +0000623 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000624 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000625 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
626 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass0750bae2022-03-10 06:00:34 +0000627 Run Keyword If ${has_dataplane} Check Ping Result True ${ping_output}
Hardik Windlassd19af862021-08-12 11:40:20 +0000628 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000629 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
630 Verify Control Plane After Pod Restart DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000631
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000632Verify restart openolt-adapter container for DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000633 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
634 ... Run the ping continuously in background during container restart,
635 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000636 ... Also, verify that the voltha control plane functionality is not affected.
637 [Tags] functionalDt RestartOpenOltPingDt
Hardik Windlassa04efa52021-08-13 08:17:24 +0000638 [Setup] Start Logging RestartOpenOltPingDt
639 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
640 ... AND Stop Logging RestartOpenOltPingDt
641 Clear All Devices Then Create New Device
642 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
643 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000644 Perform Sanity Test DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000645 FOR ${I} IN RANGE 0 ${num_all_onus}
646 ${src}= Set Variable ${hosts.src[${I}]}
647 ${dst}= Set Variable ${hosts.dst[${I}]}
648 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
649 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000650 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000651 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
652 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
653 END
654 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
655 Log ${podStatusOutput}
656 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
657 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
658 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
659 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
660 ... app ${podName} Running
661 # Wait for 1 min after openolt adapter is restarted
662 Sleep 60s
663 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
664 Log ${podStatusOutput}
665 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
666 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
667 FOR ${I} IN RANGE 0 ${num_all_onus}
668 ${src}= Set Variable ${hosts.src[${I}]}
669 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa04efa52021-08-13 08:17:24 +0000670 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000671 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000672 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
673 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000674 END
675 FOR ${I} IN RANGE 0 ${num_all_onus}
676 ${src}= Set Variable ${hosts.src[${I}]}
677 ${dst}= Set Variable ${hosts.dst[${I}]}
678 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassa04efa52021-08-13 08:17:24 +0000679 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000680 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000681 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
682 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass0750bae2022-03-10 06:00:34 +0000683 Run Keyword If ${has_dataplane} Check Ping Result True ${ping_output}
Hardik Windlassa04efa52021-08-13 08:17:24 +0000684 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000685 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
686 Verify Control Plane After Pod Restart DT
687
688Verify restart rw-core container for DT
689 [Documentation] Restart rw-core container after VOLTHA is operational.
690 ... Run the ping continuously in background during container restart,
691 ... and verify that there should be no affect on the dataplane.
692 ... Also, verify that the voltha control plane functionality is not affected.
693 [Tags] functionalDt RestartRwCorePingDt
694 [Setup] Start Logging RestartRwCorePingDt
695 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
696 ... AND Stop Logging RestartRwCorePingDt
697 Clear All Devices Then Create New Device
698 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
699 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000700 Perform Sanity Test DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000701 FOR ${I} IN RANGE 0 ${num_all_onus}
702 ${src}= Set Variable ${hosts.src[${I}]}
703 ${dst}= Set Variable ${hosts.dst[${I}]}
704 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
705 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000706 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000707 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
708 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
709 END
710 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
711 Log ${podStatusOutput}
712 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
713 ${podName} Set Variable rw-core
714 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
715 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
716 ... app ${podName} Running
717 # Wait for 1 min after rw-core is restarted
718 Sleep 60s
719 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
720 # so restart the port forwarding for the API service
721 Restart VOLTHA Port Forward voltha-api
722 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
723 Log ${podStatusOutput}
724 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
725 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
726 FOR ${I} IN RANGE 0 ${num_all_onus}
727 ${src}= Set Variable ${hosts.src[${I}]}
728 ${dst}= Set Variable ${hosts.dst[${I}]}
729 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000730 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000731 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
732 ... ${src['container_type']} ${src['container_name']}
733 END
734 FOR ${I} IN RANGE 0 ${num_all_onus}
735 ${src}= Set Variable ${hosts.src[${I}]}
736 ${dst}= Set Variable ${hosts.dst[${I}]}
737 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
738 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000739 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000740 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
741 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass0750bae2022-03-10 06:00:34 +0000742 Run Keyword If ${has_dataplane} Check Ping Result True ${ping_output}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000743 END
744 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
745 Verify Control Plane After Pod Restart DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000746
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530747*** Keywords ***
748Setup Suite
749 [Documentation] Set up the test suite
750 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200751 #power_switch.robot needs it to support different vendor's power switch
752 ${switch_type}= Get Variable Value ${web_power_switch.type}
753 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000754 # Run Pre-test Setup for Soak Job
755 # Note: As soak requirement, it expects that the devices under test are already created and enabled
756 Run Keyword If '${SOAK_TEST}'=='True' Setup Soak
757
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530758
759Clear All Devices Then Create New Device
760 [Documentation] Remove any devices from VOLTHA and ONOS
761 # Remove all devices from voltha and nos
762 Delete All Devices and Verify
763 # Execute normal test Setup Keyword
764 Setup
765
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000766Verify Control Plane After Pod Restart DT
767 [Documentation] Verifies the control plane functionality after the voltha pod restart
768 ... by deleting and re-adding the subscriber
769 FOR ${I} IN RANGE 0 ${num_all_onus}
770 ${src}= Set Variable ${hosts.src[${I}]}
771 ${dst}= Set Variable ${hosts.dst[${I}]}
772 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000773 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000774 ${onu_device_id}= Get Device ID From SN ${src['onu']}
775 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Emrehan UZUNabef5182021-11-26 10:54:24 +0000776 ... ${of_id} ${src['uni_id']}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000777 # Remove Subscriber Access
778 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
779 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
780 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000781 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000782 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
783 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
784 # Disable and Re-Enable the ONU (To replicate DT current workflow)
785 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
786 Disable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000787 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000788 ... Validate Device DISABLED UNKNOWN
789 ... REACHABLE ${src['onu']}
790 Enable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000791 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000792 ... Validate Device ENABLED ACTIVE
793 ... REACHABLE ${src['onu']}
794 # Add Subscriber Access
795 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
796 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
797 # Verify subscriber access flows are added for the ONU port
798 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
799 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
800 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000801 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000802 ... Validate Device ENABLED ACTIVE
803 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
Girish Gowdra8fd96962021-12-14 16:20:04 +0530804 # Workaround for issue seen in VOL-4489. Keep this workaround until VOL-4489 is fixed.
805 Run Keyword If ${has_dataplane} Reboot XGSPON ONU ${src['olt']} ${src['onu']} omci-flows-pushed
806 # Workaround ends here for issue seen in VOL-4489.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000807 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000808 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000809 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
810 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
811 END