blob: 2b6635b25d9f07aaf4de63595df90c226104a8c9 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
Hardik Windlassb9bdd502020-04-17 14:46:21 +053047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
54${teardown_device} False
55${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Hardik Windlass43531332021-05-19 07:07:03 +000060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
Hardik Windlass4ed54502021-05-14 07:34:15 +000064# Flag specific to Soak Jobs
65${SOAK_TEST} False
66
Hardik Windlassb9bdd502020-04-17 14:46:21 +053067*** Test Cases ***
68Verify ONU after Rebooting Physically for DT
69 [Documentation] Test the ONU functionality by physically turning on/off ONU.
70 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
71 ... Test case runs only on the PODs that are configured with PowerSwitch that
72 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
73 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053074 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlass43531332021-05-19 07:07:03 +000075 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070077 ... AND Run Keyword If ${logging} Collect Logs
78 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053079 # Add OLT device
80 Setup
81 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
82 Run Keyword If ${has_dataplane} Clean Up Linux
83 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
84 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053085 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080086 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053087 ${src}= Set Variable ${hosts.src[${I}]}
88 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080089 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass0560f802020-05-06 23:06:53 +053090 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053091 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
92 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053093 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053094 Disable Switch Outlet ${src['power_switch_port']}
95 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
96 ... Wait Until Keyword Succeeds 60s 2s
97 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
98 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053099 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000100 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530101 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Girish Gowdrae0c711e2021-07-20 14:30:17 -0700102 # Allow the remove subscriber command to clean up all the flows, schedulers and queues before deleting the device
103 Sleep 5s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530104 # Delete ONU Device (To replicate DT workflow)
105 Delete Device ${onu_device_id}
106 Sleep 5s
107 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530108 Enable Switch Outlet ${src['power_switch_port']}
Hardik Windlass40ccffb2020-05-11 14:12:07 +0530109 # Waiting extra time for the ONU to come up
110 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530111 # Check ONU port is Enabled in ONOS
112 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000113 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530114 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000115 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530116 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530117 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700118 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530119 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530120 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530121 # Verify subscriber access flows are added for the ONU port
122 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
123 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
124 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530125 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530126 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
127 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530128 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530129 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
130 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530131 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800132 # Verify flows for all OLTs
133 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530134
135Verify OLT after Rebooting Physically for DT
136 [Documentation] Test the physical reboot of the OLT
137 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
138 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
139 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530140 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000141 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800142 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700143 ... AND Run Keyword If ${logging} Collect Logs
144 ... AND Stop Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530145 # Add OLT device
146 Setup
147 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
148 Run Keyword If ${has_dataplane} Clean Up Linux
149 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
150 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800151 FOR ${I} IN RANGE 0 ${olt_count}
152 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
153 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
154 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
155 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
156 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
157 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100158 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800159 END
160 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530161 ${src}= Set Variable ${hosts.src[${I}]}
162 ${dst}= Set Variable ${hosts.dst[${I}]}
163 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
164 ... Wait Until Keyword Succeeds 60s 2s
165 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
166 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
167 END
168 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800169 FOR ${I} IN RANGE 0 ${olt_count}
170 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
171 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
172 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
173 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
174 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
175 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
176 ... Check Remote System Reachability True ${olt_ssh_ip}
177 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
178 ... Validate OLT Device ENABLED ACTIVE
179 ... REACHABLE ${olt_serial_number}
180 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530181 # Waiting extra time for the ONUs to come up
182 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530183 Run Keyword If ${has_dataplane} Clean Up Linux
184 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
185
TorstenThiemeed3332f2021-02-02 10:30:48 +0000186Verify restart openonu-adapter container after subscriber provisioning for DT
187 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
188 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000189 [Tags] Restart-OpenOnu-Dt soak
TorstenThiemeed3332f2021-02-02 10:30:48 +0000190 [Setup] Start Logging Restart-OpenOnu-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000191 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
TorstenThiemeed3332f2021-02-02 10:30:48 +0000192 ... AND Stop Logging Restart-OpenOnu-Dt
TorstenThiemeed3332f2021-02-02 10:30:48 +0000193 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000194 Run Keyword If '${SOAK_TEST}'=='False' Setup
TorstenThiemeed3332f2021-02-02 10:30:48 +0000195 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
196 Run Keyword If ${has_dataplane} Clean Up Linux
197 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
198 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
199 Log ${podStatusOutput}
200 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
201 ${podName} Set Variable adapter-open-onu
202 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000203 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
204 ... app ${podName} Running
205 # Wait for 1 min after openonu adapter is restarted
206 Sleep 60s
207 Run Keyword If ${has_dataplane} Clean Up Linux
208 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000209 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
210 Log ${podStatusOutput}
211 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
212 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
213 Log to console Pod ${podName} restarted and sanity checks passed successfully
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000214 Run Keyword If '${SOAK_TEST}'=='False' Delete All Devices and Verify
TorstenThiemeed3332f2021-02-02 10:30:48 +0000215
Hardik Windlass9df139e2020-04-24 14:54:54 +0530216Verify restart openolt-adapter container after subscriber provisioning for DT
217 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
218 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000219 [Tags] Restart-OpenOlt-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530220 [Setup] Start Logging Restart-OpenOlt-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000221 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530222 ... AND Stop Logging Restart-OpenOlt-Dt
223 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000224 Run Keyword If '${SOAK_TEST}'=='False' setup
Hardik Windlass9df139e2020-04-24 14:54:54 +0530225 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
226 Run Keyword If ${has_dataplane} Clean Up Linux
227 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530228 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
229 Log ${podStatusOutput}
230 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100231 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530232 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530233 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
234 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000235 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530236 Sleep 60s
237 Run Keyword If ${has_dataplane} Clean Up Linux
238 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530239 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
240 Log ${podStatusOutput}
241 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
242 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
243 Log to console Pod ${podName} restarted and sanity checks passed successfully
244
245Verify openolt adapter restart before subscriber provisioning for DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000246 [Documentation] Restart openolt-adapter container before adding the subscriber.
247 [Tags] functionalDt olt-adapter-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530248 [Setup] Start Logging OltAdapterRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000249 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530250 ... AND Stop Logging OltAdapterRestart-Dt
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000251 # Add OLT device
252 Clear All Devices Then Create New Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530253 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass9df139e2020-04-24 14:54:54 +0530254 Set Global Variable ${of_id}
255
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800256 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530257 ${src}= Set Variable ${hosts.src[${I}]}
258 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800259 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530260 ${onu_device_id}= Get Device ID From SN ${src['onu']}
261 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
262 ... ${of_id}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000263 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
264 ... Validate Device ENABLED ACTIVE REACHABLE
265 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530266 END
267 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100268 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
269 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
270 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530271 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100272 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530273 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100274 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530275
276 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800277 FOR ${I} IN RANGE 0 ${olt_count}
278 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
279 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
280 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
281 ... ${olt_serial_number}
282 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
283 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
284 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530285
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800286 FOR ${I} IN RANGE 0 ${num_all_onus}
287 ${src}= Set Variable ${hosts.src[${I}]}
288 ${dst}= Set Variable ${hosts.dst[${I}]}
289 ${of_id}= Get ofID From OLT List ${src['olt']}
290 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
291 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530292 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000293 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530294 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
295 # Verify subscriber access flows are added for the ONU port
296 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
297 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
298 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000299 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
300 ... ENABLED ACTIVE REACHABLE
301 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass9df139e2020-04-24 14:54:54 +0530302 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
303 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
304 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
305 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
306 ... ${dst['container_name']}
307 END
308
309Verify restart ofagent container after subscriber is provisioned for DT
310 [Documentation] Restart ofagent container after VOLTHA is operational.
311 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000312 [Tags] functionalDt ofagentRestart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530313 [Setup] Start Logging ofagentRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000314 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530315 ... AND Stop Logging ofagentRestart-Dt
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700316 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530317 # set timeout value
318 ${waitforRestart} Set Variable 120s
319 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
320 Log ${podStatusOutput}
321 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
322 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530323 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530324 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
325 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000326 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530327 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
328 Run Keyword If ${has_dataplane} Clean Up Linux
329 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
330 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
331 Log ${podStatusOutput}
332 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
333 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
334 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700335 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530336 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800337 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530338 ${src}= Set Variable ${hosts.src[${I}]}
339 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800340 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530341 ${onu_device_id}= Get Device ID From SN ${src['onu']}
342 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
343 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
344 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700345 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530346 ... ENABLED ACTIVE REACHABLE
347 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
348 # Check ONU port is Disabled in ONOS
349 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000350 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530351 # Verify subscriber access flows are added for the ONU port
352 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
353 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
354 ... ${onu_port} ${nni_port} ${src['s_tag']}
355 # Verify Ping
356 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
357 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
358 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530359 END
360 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700361 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530362 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
363 ... Running
364 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
365 Run Keyword If ${has_dataplane} Clean Up Linux
366 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
367 Log to console Pod ${podName} restarted and sanity checks passed successfully
368
369Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
370 [Documentation] Deploys an device instance and waits for it to authenticate. After
371 ... authentication is successful the rw-core deployment is scaled to 0 instances to
372 ... simulate a POD crash. The test then scales the rw-core back to a single instance
373 ... and configures ONOS for access. The test succeeds if the device is able to
374 ... complete the DHCP sequence.
Hardik Windlass43531332021-05-19 07:07:03 +0000375 [Tags] functionalDt rwcore-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530376 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
377 ... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000378 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530379 ... AND Stop Logging RwCoreFailAndRestart-Dt
380 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530381 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800382 FOR ${I} IN RANGE 0 ${olt_count}
383 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
384 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
385 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
386 ... ${olt_serial_number}
387 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
388 ... Get NNI Port in ONOS ${of_id}
389 # Set Global Variable ${nni_port}
390 END
391 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530392 ${src}= Set Variable ${hosts.src[${I}]}
393 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800394 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530395 ${onu_device_id}= Get Device ID From SN ${src['onu']}
396 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
397 ... ${of_id}
398 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700399 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000400 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530401 END
402
403 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530404 Scale K8s Deployment voltha voltha-voltha-rw-core 0
405 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530406 # Ensure the ofagent POD goes "not-ready" as expected
407 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530408 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530409 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530410 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530411 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530412 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530413 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530414 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530415 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
416 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700417 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530418 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
419 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800420 FOR ${I} IN RANGE 0 ${olt_count}
421 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
422 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
423 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
424 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700425 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800426 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
427 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530428
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800429 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700430 ${src}= Set Variable ${hosts.src[${I}]}
431 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800432 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavier52094622020-05-12 15:54:24 -0700433 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
434 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530435 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000436 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530437 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
438 # Verify subscriber access flows are added for the ONU port
439 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
440 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
441 ... ${onu_port} ${nni_port} ${src['s_tag']}
442 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
443 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
444 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
445 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
446 ... ${dst['container_name']}
447 END
448
Hemadf003682020-04-28 21:22:22 +0530449Verify OLT Soft Reboot for DT
450 [Documentation] Test soft reboot of the OLT using voltctl command
Hardik Windlass20a503d2021-06-21 11:58:08 +0000451 [Tags] VOL-2818 OLTSoftRebootDt functionalDt
Hemadf003682020-04-28 21:22:22 +0530452 [Setup] Start Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000453 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hemadf003682020-04-28 21:22:22 +0530454 ... AND Stop Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000455 FOR ${I} IN RANGE 0 ${olt_count}
456 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
457 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
458 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
459 ... Validate OLT Device ENABLED ACTIVE
460 ... REACHABLE ${olt_serial_number}
461 # Reboot the OLT using "voltctl device reboot" command
462 Reboot Device ${olt_device_id}
463 # Wait for the OLT to actually go down
464 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
465 ... ${olt_serial_number}
466 END
Hemadf003682020-04-28 21:22:22 +0530467 #Verify that ping fails
Hardik Windlass20a503d2021-06-21 11:58:08 +0000468 FOR ${I} IN RANGE 0 ${num_all_onus}
Hemadf003682020-04-28 21:22:22 +0530469 ${src}= Set Variable ${hosts.src[${I}]}
470 ${dst}= Set Variable ${hosts.dst[${I}]}
471 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
472 ... Wait Until Keyword Succeeds 60s 2s
473 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
474 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
475 END
Hardik Windlass20a503d2021-06-21 11:58:08 +0000476 # Check OLT states
477 FOR ${I} IN RANGE 0 ${olt_count}
478 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
479 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
480 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
481 # Wait for the OLT to come back up
482 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
483 ... Check Remote System Reachability True ${olt_ssh_ip}
484 # Check OLT states
485 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
486 ... Validate OLT Device ENABLED ACTIVE
487 ... REACHABLE ${olt_serial_number}
488 END
Hemadf003682020-04-28 21:22:22 +0530489 # Waiting extra time for the ONUs to come up
490 Sleep 60s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000491 #Check after reboot that ONUs are active, DHCP/pingable
Hemadf003682020-04-28 21:22:22 +0530492 Run Keyword If ${has_dataplane} Clean Up Linux
493 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
494
Hema8c1a0322020-05-26 16:02:32 +0530495Verify ONU Soft Reboot for DT
496 [Documentation] Test soft reboot of the ONU using voltctl command
497 [Tags] VOL-2820 ONUSoftRebootDt notready
498 [Setup] Start Logging ONUSoftRebootDt
499 #... AND Setup
500 [Teardown] Run Keywords Collect Logs
501 ... AND Stop Logging ONUSoftRebootDt
502 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530503 #Reboot the ONU and verify that ping fails
504 FOR ${I} IN RANGE 0 ${num_onus}
505 ${src}= Set Variable ${hosts.src[${I}]}
506 ${dst}= Set Variable ${hosts.dst[${I}]}
507 ${onu_device_id}= Get Device ID From SN ${src['onu']}
508 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
509 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
510 Reboot Device ${onu_device_id}
511 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
512 ... Wait Until Keyword Succeeds 60s 2s
513 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
514 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
515 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
516 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000517 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hema8c1a0322020-05-26 16:02:32 +0530518 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
519 # Delete ONU Device (To replicate DT workflow)
520 Delete Device ${onu_device_id}
521 Sleep 40s
522 # Check ONU port is Enabled in ONOS
523 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000524 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530525 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000526 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hema8c1a0322020-05-26 16:02:32 +0530527 ... volt-add-subscriber-access ${of_id} ${onu_port}
528 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700529 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530530 ... ENABLED ACTIVE REACHABLE
531 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
532 # Verify subscriber access flows are added for the ONU port
533 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
534 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
535 ... ${onu_port} ${nni_port} ${src['s_tag']}
536 #Run Keyword If ${has_dataplane} Clean Up Linux
537 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
538 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
539 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
540 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
541 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530542 END
543 # Verify ONOS Flows
544 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
545 ${onos_flows_count}= Evaluate 4 * ${num_onus}
546 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
547 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
548 ... ${of_id} ${onos_flows_count}
549 # Verify VOLTHA Flows
550 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
551 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
552 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
553 ${onu_flows}= Set Variable 2
554 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
555 ${List_ONU_Serial} Create List
556 Set Suite Variable ${List_ONU_Serial}
557 Build ONU SN List ${List_ONU_Serial}
558 Log ${List_ONU_Serial}
559 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
560 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530561
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000562Verify restart openonu-adapter container for DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000563 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
564 ... Run the ping continuously in background during container restart,
565 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000566 ... Also, verify that the voltha control plane functionality is not affected.
567 [Tags] functionalDt RestartOpenOnuPingDt
Hardik Windlassd19af862021-08-12 11:40:20 +0000568 [Setup] Start Logging RestartOpenOnuPingDt
569 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
570 ... AND Stop Logging RestartOpenOnuPingDt
571 Clear All Devices Then Create New Device
572 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
573 Run Keyword If ${has_dataplane} Clean Up Linux
574 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
575 FOR ${I} IN RANGE 0 ${num_all_onus}
576 ${src}= Set Variable ${hosts.src[${I}]}
577 ${dst}= Set Variable ${hosts.dst[${I}]}
578 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
579 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
580 ... Wait Until Keyword Succeeds 60s 2s
581 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
582 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
583 END
584 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
585 Log ${podStatusOutput}
586 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
587 ${podName} Set Variable adapter-open-onu
588 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
589 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
590 ... app ${podName} Running
591 # Wait for 1 min after openonu adapter is restarted
592 Sleep 60s
593 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
594 Log ${podStatusOutput}
595 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
596 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
597 FOR ${I} IN RANGE 0 ${num_all_onus}
598 ${src}= Set Variable ${hosts.src[${I}]}
599 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassd19af862021-08-12 11:40:20 +0000600 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
601 ... Wait Until Keyword Succeeds 60s 2s
602 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
603 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000604 END
605 FOR ${I} IN RANGE 0 ${num_all_onus}
606 ${src}= Set Variable ${hosts.src[${I}]}
607 ${dst}= Set Variable ${hosts.dst[${I}]}
608 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassd19af862021-08-12 11:40:20 +0000609 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
610 ... Wait Until Keyword Succeeds 60s 2s
611 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
612 ... ${src['container_type']} ${src['container_name']}
613 Check Ping Result True ${ping_output}
614 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000615 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
616 Verify Control Plane After Pod Restart DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000617
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000618Verify restart openolt-adapter container for DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000619 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
620 ... Run the ping continuously in background during container restart,
621 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000622 ... Also, verify that the voltha control plane functionality is not affected.
623 [Tags] functionalDt RestartOpenOltPingDt
Hardik Windlassa04efa52021-08-13 08:17:24 +0000624 [Setup] Start Logging RestartOpenOltPingDt
625 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
626 ... AND Stop Logging RestartOpenOltPingDt
627 Clear All Devices Then Create New Device
628 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
629 Run Keyword If ${has_dataplane} Clean Up Linux
630 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
631 FOR ${I} IN RANGE 0 ${num_all_onus}
632 ${src}= Set Variable ${hosts.src[${I}]}
633 ${dst}= Set Variable ${hosts.dst[${I}]}
634 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
635 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
636 ... Wait Until Keyword Succeeds 60s 2s
637 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
638 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
639 END
640 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
641 Log ${podStatusOutput}
642 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
643 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
644 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
645 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
646 ... app ${podName} Running
647 # Wait for 1 min after openolt adapter is restarted
648 Sleep 60s
649 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
650 Log ${podStatusOutput}
651 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
652 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
653 FOR ${I} IN RANGE 0 ${num_all_onus}
654 ${src}= Set Variable ${hosts.src[${I}]}
655 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa04efa52021-08-13 08:17:24 +0000656 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
657 ... Wait Until Keyword Succeeds 60s 2s
658 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
659 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000660 END
661 FOR ${I} IN RANGE 0 ${num_all_onus}
662 ${src}= Set Variable ${hosts.src[${I}]}
663 ${dst}= Set Variable ${hosts.dst[${I}]}
664 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassa04efa52021-08-13 08:17:24 +0000665 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
666 ... Wait Until Keyword Succeeds 60s 2s
667 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
668 ... ${src['container_type']} ${src['container_name']}
669 Check Ping Result True ${ping_output}
670 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000671 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
672 Verify Control Plane After Pod Restart DT
673
674Verify restart rw-core container for DT
675 [Documentation] Restart rw-core container after VOLTHA is operational.
676 ... Run the ping continuously in background during container restart,
677 ... and verify that there should be no affect on the dataplane.
678 ... Also, verify that the voltha control plane functionality is not affected.
679 [Tags] functionalDt RestartRwCorePingDt
680 [Setup] Start Logging RestartRwCorePingDt
681 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
682 ... AND Stop Logging RestartRwCorePingDt
683 Clear All Devices Then Create New Device
684 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
685 Run Keyword If ${has_dataplane} Clean Up Linux
686 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
687 FOR ${I} IN RANGE 0 ${num_all_onus}
688 ${src}= Set Variable ${hosts.src[${I}]}
689 ${dst}= Set Variable ${hosts.dst[${I}]}
690 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
691 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
692 ... Wait Until Keyword Succeeds 60s 2s
693 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
694 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
695 END
696 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
697 Log ${podStatusOutput}
698 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
699 ${podName} Set Variable rw-core
700 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
701 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
702 ... app ${podName} Running
703 # Wait for 1 min after rw-core is restarted
704 Sleep 60s
705 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
706 # so restart the port forwarding for the API service
707 Restart VOLTHA Port Forward voltha-api
708 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
709 Log ${podStatusOutput}
710 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
711 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
712 FOR ${I} IN RANGE 0 ${num_all_onus}
713 ${src}= Set Variable ${hosts.src[${I}]}
714 ${dst}= Set Variable ${hosts.dst[${I}]}
715 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
716 ... Wait Until Keyword Succeeds 60s 2s
717 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
718 ... ${src['container_type']} ${src['container_name']}
719 END
720 FOR ${I} IN RANGE 0 ${num_all_onus}
721 ${src}= Set Variable ${hosts.src[${I}]}
722 ${dst}= Set Variable ${hosts.dst[${I}]}
723 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
724 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
725 ... Wait Until Keyword Succeeds 60s 2s
726 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
727 ... ${src['container_type']} ${src['container_name']}
728 Check Ping Result True ${ping_output}
729 END
730 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
731 Verify Control Plane After Pod Restart DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000732
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530733*** Keywords ***
734Setup Suite
735 [Documentation] Set up the test suite
736 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200737 #power_switch.robot needs it to support different vendor's power switch
738 ${switch_type}= Get Variable Value ${web_power_switch.type}
739 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000740 # Run Pre-test Setup for Soak Job
741 # Note: As soak requirement, it expects that the devices under test are already created and enabled
742 Run Keyword If '${SOAK_TEST}'=='True' Setup Soak
743
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530744
745Clear All Devices Then Create New Device
746 [Documentation] Remove any devices from VOLTHA and ONOS
747 # Remove all devices from voltha and nos
748 Delete All Devices and Verify
749 # Execute normal test Setup Keyword
750 Setup
751
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000752Verify Control Plane After Pod Restart DT
753 [Documentation] Verifies the control plane functionality after the voltha pod restart
754 ... by deleting and re-adding the subscriber
755 FOR ${I} IN RANGE 0 ${num_all_onus}
756 ${src}= Set Variable ${hosts.src[${I}]}
757 ${dst}= Set Variable ${hosts.dst[${I}]}
758 ${of_id}= Get ofID From OLT List ${src['olt']}
759 ${onu_device_id}= Get Device ID From SN ${src['onu']}
760 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
761 ... ${of_id}
762 # Remove Subscriber Access
763 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
764 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
765 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
766 ... Wait Until Keyword Succeeds 60s 2s
767 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
768 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
769 # Disable and Re-Enable the ONU (To replicate DT current workflow)
770 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
771 Disable Device ${onu_device_id}
772 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
773 ... Validate Device DISABLED UNKNOWN
774 ... REACHABLE ${src['onu']}
775 Enable Device ${onu_device_id}
776 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
777 ... Validate Device ENABLED ACTIVE
778 ... REACHABLE ${src['onu']}
779 # Add Subscriber Access
780 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
781 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
782 # Verify subscriber access flows are added for the ONU port
783 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
784 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
785 ... ${onu_port} ${nni_port} ${src['s_tag']}
786 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
787 ... Validate Device ENABLED ACTIVE
788 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
789 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
790 ... Wait Until Keyword Succeeds 60s 2s
791 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
792 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
793 END