blob: 92ad0265c9aaedbd49e5b9a5ca8019f25cbbf948 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
45# For below variable value, using deployment name as using grep for
46# parsing radius pod name, we can also use full radius pod name
47${RESTART_POD_NAME} radius
48${timeout} 60s
49${of_id} 0
50${logical_id} 0
51${has_dataplane} True
52${teardown_device} False
53${scripts} ../../scripts
54
55# Per-test logging on failure is turned off by default; set this variable to enable
56${container_log_dir} ${None}
57
Hardik Windlass43531332021-05-19 07:07:03 +000058# logging flag to enable Collect Logs, can be passed via the command line too
59# example: -v logging:False
60${logging} True
61
Hardik Windlass4ed54502021-05-14 07:34:15 +000062# Flag specific to Soak Jobs
63${SOAK_TEST} False
64
Hardik Windlassb9bdd502020-04-17 14:46:21 +053065*** Test Cases ***
66Verify ONU after Rebooting Physically for DT
67 [Documentation] Test the ONU functionality by physically turning on/off ONU.
68 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
69 ... Test case runs only on the PODs that are configured with PowerSwitch that
70 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
71 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053072 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlass43531332021-05-19 07:07:03 +000073 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlassb9bdd502020-04-17 14:46:21 +053074 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080075 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +053076 # Add OLT device
77 Setup
78 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
79 Run Keyword If ${has_dataplane} Clean Up Linux
80 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
81 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053082 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080083 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 ${src}= Set Variable ${hosts.src[${I}]}
85 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080086 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass0560f802020-05-06 23:06:53 +053087 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053088 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
89 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053090 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053091 Disable Switch Outlet ${src['power_switch_port']}
92 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
93 ... Wait Until Keyword Succeeds 60s 2s
94 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
95 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +053096 # Remove Subscriber Access (To replicate DT workflow)
97 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
98 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
99 # Delete ONU Device (To replicate DT workflow)
100 Delete Device ${onu_device_id}
101 Sleep 5s
102 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530103 Enable Switch Outlet ${src['power_switch_port']}
Hardik Windlass40ccffb2020-05-11 14:12:07 +0530104 # Waiting extra time for the ONU to come up
105 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530106 # Check ONU port is Enabled in ONOS
107 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700108 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530109 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
110 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
111 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530112 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700113 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530114 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530115 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530116 # Verify subscriber access flows are added for the ONU port
117 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
118 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
119 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530120 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530121 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
122 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530123 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530124 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
125 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530126 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800127 # Verify flows for all OLTs
128 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530129
130Verify OLT after Rebooting Physically for DT
131 [Documentation] Test the physical reboot of the OLT
132 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
133 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
134 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530135 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000136 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530137 ... AND Stop Logging RebootOlt_Physical_Dt
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800138 ... AND Delete All Devices and Verify
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530139 # Add OLT device
140 Setup
141 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
142 Run Keyword If ${has_dataplane} Clean Up Linux
143 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
144 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800145 FOR ${I} IN RANGE 0 ${olt_count}
146 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
147 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
148 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
149 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
150 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
151 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100152 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800153 END
154 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530155 ${src}= Set Variable ${hosts.src[${I}]}
156 ${dst}= Set Variable ${hosts.dst[${I}]}
157 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
158 ... Wait Until Keyword Succeeds 60s 2s
159 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
160 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
161 END
162 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800163 FOR ${I} IN RANGE 0 ${olt_count}
164 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
165 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
166 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
167 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
168 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
169 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
170 ... Check Remote System Reachability True ${olt_ssh_ip}
171 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
172 ... Validate OLT Device ENABLED ACTIVE
173 ... REACHABLE ${olt_serial_number}
174 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530175 # Waiting extra time for the ONUs to come up
176 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530177 Run Keyword If ${has_dataplane} Clean Up Linux
178 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
179
TorstenThiemeed3332f2021-02-02 10:30:48 +0000180Verify restart openonu-adapter container after subscriber provisioning for DT
181 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
182 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass43531332021-05-19 07:07:03 +0000183 [Tags] functionalDt Restart-OpenOnu-Dt soak
TorstenThiemeed3332f2021-02-02 10:30:48 +0000184 [Setup] Start Logging Restart-OpenOnu-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000185 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
TorstenThiemeed3332f2021-02-02 10:30:48 +0000186 ... AND Stop Logging Restart-OpenOnu-Dt
TorstenThiemeed3332f2021-02-02 10:30:48 +0000187 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000188 Run Keyword If '${SOAK_TEST}'=='False' Setup
TorstenThiemeed3332f2021-02-02 10:30:48 +0000189 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
190 Run Keyword If ${has_dataplane} Clean Up Linux
191 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
192 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
193 Log ${podStatusOutput}
194 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
195 ${podName} Set Variable adapter-open-onu
196 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000197 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
198 ... app ${podName} Running
199 # Wait for 1 min after openonu adapter is restarted
200 Sleep 60s
201 Run Keyword If ${has_dataplane} Clean Up Linux
202 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000203 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
204 Log ${podStatusOutput}
205 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
206 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
207 Log to console Pod ${podName} restarted and sanity checks passed successfully
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000208 Run Keyword If '${SOAK_TEST}'=='False' Delete All Devices and Verify
TorstenThiemeed3332f2021-02-02 10:30:48 +0000209
Hardik Windlass9df139e2020-04-24 14:54:54 +0530210Verify restart openolt-adapter container after subscriber provisioning for DT
211 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
212 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass43531332021-05-19 07:07:03 +0000213 [Tags] functionalDt Restart-OpenOlt-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530214 [Setup] Start Logging Restart-OpenOlt-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000215 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530216 ... AND Stop Logging Restart-OpenOlt-Dt
217 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000218 Run Keyword If '${SOAK_TEST}'=='False' setup
Hardik Windlass9df139e2020-04-24 14:54:54 +0530219 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
220 Run Keyword If ${has_dataplane} Clean Up Linux
221 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530222 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
223 Log ${podStatusOutput}
224 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100225 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530226 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530227 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
228 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000229 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530230 Sleep 60s
231 Run Keyword If ${has_dataplane} Clean Up Linux
232 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530233 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
234 Log ${podStatusOutput}
235 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
236 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
237 Log to console Pod ${podName} restarted and sanity checks passed successfully
238
239Verify openolt adapter restart before subscriber provisioning for DT
240 [Documentation] Deploys an device instance and waits for it to authenticate. After
241 ... authentication is successful the rw-core deployment is scaled to 0 instances to
242 ... simulate a POD crash. The test then scales the rw-core back to a single instance
243 ... and configures ONOS for access. The test succeeds if the device is able to
244 ... complete the DHCP sequence.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000245 [Tags] functionalDt olt-adapter-restart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530246 [Setup] Start Logging OltAdapterRestart-Dt
247 #... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000248 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530249 ... AND Stop Logging OltAdapterRestart-Dt
250 # Add OLT and perform sanity test
251 #setup
252 Run Keyword If ${has_dataplane} Clean Up Linux
253 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
254 Set Global Variable ${of_id}
255
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800256 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530257 ${src}= Set Variable ${hosts.src[${I}]}
258 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800259 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530260 ${onu_device_id}= Get Device ID From SN ${src['onu']}
261 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
262 ... ${of_id}
263 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700264 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530265 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
266 END
267 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100268 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
269 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
270 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530271 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100272 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530273 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100274 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530275
276 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800277 FOR ${I} IN RANGE 0 ${olt_count}
278 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
279 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
280 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
281 ... ${olt_serial_number}
282 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
283 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
284 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530285
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800286 FOR ${I} IN RANGE 0 ${num_all_onus}
287 ${src}= Set Variable ${hosts.src[${I}]}
288 ${dst}= Set Variable ${hosts.dst[${I}]}
289 ${of_id}= Get ofID From OLT List ${src['olt']}
290 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
291 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530292 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
293 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
294 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
295 # Verify subscriber access flows are added for the ONU port
296 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
297 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
298 ... ${onu_port} ${nni_port} ${src['s_tag']}
299 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
300 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
301 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
302 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
303 ... ${dst['container_name']}
304 END
305
306Verify restart ofagent container after subscriber is provisioned for DT
307 [Documentation] Restart ofagent container after VOLTHA is operational.
308 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000309 [Tags] functionalDt ofagentRestart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530310 [Setup] Start Logging ofagentRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000311 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530312 ... AND Stop Logging ofagentRestart-Dt
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530313 ... AND Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530314 # set timeout value
315 ${waitforRestart} Set Variable 120s
316 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
317 Log ${podStatusOutput}
318 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
319 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530320 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530321 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
322 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000323 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530324 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
325 Run Keyword If ${has_dataplane} Clean Up Linux
326 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
327 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
328 Log ${podStatusOutput}
329 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
330 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
331 # Scale Down the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530332 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530333 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800334 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530335 ${src}= Set Variable ${hosts.src[${I}]}
336 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800337 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530338 ${onu_device_id}= Get Device ID From SN ${src['onu']}
339 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
340 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
341 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700342 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530343 ... ENABLED ACTIVE REACHABLE
344 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
345 # Check ONU port is Disabled in ONOS
346 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700347 ... Verify ONU Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530348 # Verify subscriber access flows are added for the ONU port
349 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
350 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
351 ... ${onu_port} ${nni_port} ${src['s_tag']}
352 # Verify Ping
353 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
354 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
355 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530356 END
357 # Scale Up the Of-Agent Deployment
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530358 Scale K8s Deployment ${NAMESPACE} voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530359 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
360 ... Running
361 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
362 Run Keyword If ${has_dataplane} Clean Up Linux
363 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
364 Log to console Pod ${podName} restarted and sanity checks passed successfully
365
366Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
367 [Documentation] Deploys an device instance and waits for it to authenticate. After
368 ... authentication is successful the rw-core deployment is scaled to 0 instances to
369 ... simulate a POD crash. The test then scales the rw-core back to a single instance
370 ... and configures ONOS for access. The test succeeds if the device is able to
371 ... complete the DHCP sequence.
Hardik Windlass43531332021-05-19 07:07:03 +0000372 [Tags] functionalDt rwcore-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530373 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
374 ... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000375 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530376 ... AND Stop Logging RwCoreFailAndRestart-Dt
377 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530378 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800379 FOR ${I} IN RANGE 0 ${olt_count}
380 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
381 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
382 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
383 ... ${olt_serial_number}
384 ${nni_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
385 ... Get NNI Port in ONOS ${of_id}
386 # Set Global Variable ${nni_port}
387 END
388 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530389 ${src}= Set Variable ${hosts.src[${I}]}
390 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800391 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530392 ${onu_device_id}= Get Device ID From SN ${src['onu']}
393 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
394 ... ${of_id}
395 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700396 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass9df139e2020-04-24 14:54:54 +0530397 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded
398 END
399
400 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530401 Scale K8s Deployment voltha voltha-voltha-rw-core 0
402 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530403 # Ensure the ofagent POD goes "not-ready" as expected
404 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530405 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530406 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530407 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530408 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530409 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530410 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530411 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530412 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
413 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700414 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530415 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
416 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800417 FOR ${I} IN RANGE 0 ${olt_count}
418 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
419 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
420 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
421 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700422 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800423 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
424 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530425
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800426 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700427 ${src}= Set Variable ${hosts.src[${I}]}
428 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800429 ${of_id}= Get ofID From OLT List ${src['olt']}
Andy Bavier52094622020-05-12 15:54:24 -0700430 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
431 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530432 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
433 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
434 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
435 # Verify subscriber access flows are added for the ONU port
436 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
437 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
438 ... ${onu_port} ${nni_port} ${src['s_tag']}
439 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
440 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
441 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
442 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
443 ... ${dst['container_name']}
444 END
445
Hemadf003682020-04-28 21:22:22 +0530446Verify OLT Soft Reboot for DT
447 [Documentation] Test soft reboot of the OLT using voltctl command
448 [Tags] VOL-2818 OLTSoftRebootDt notready
449 [Setup] Start Logging OLTSoftRebootDt
450 #... AND Setup
451 [Teardown] Run Keywords Collect Logs
452 ... AND Stop Logging OLTSoftRebootDt
453 #... AND Delete Device and Verify
Hemadf003682020-04-28 21:22:22 +0530454 # Reboot the OLT using "voltctl device reboot" command
455 Reboot Device ${olt_device_id}
Hemadf003682020-04-28 21:22:22 +0530456 #Verify that ping fails
457 FOR ${I} IN RANGE 0 ${num_onus}
458 ${src}= Set Variable ${hosts.src[${I}]}
459 ${dst}= Set Variable ${hosts.dst[${I}]}
460 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
461 ... Wait Until Keyword Succeeds 60s 2s
462 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
463 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
464 END
465 # Wait for the OLT to come back up
466 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
Suchitra Vemurib7253a52020-07-14 22:35:17 -0700467 ... Check Remote System Reachability True ${olt_ssh_ip}
Hemadf003682020-04-28 21:22:22 +0530468 # Waiting extra time for the ONUs to come up
469 Sleep 60s
470 # Check OLT states
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700471 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED ACTIVE REACHABLE
Hemadf003682020-04-28 21:22:22 +0530472 ... ${olt_serial_number}
Hemadf003682020-04-28 21:22:22 +0530473 #Check after reboot that ONUs are active, DHCP and pingable
474 Run Keyword If ${has_dataplane} Clean Up Linux
475 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test DT
476
Hema8c1a0322020-05-26 16:02:32 +0530477Verify ONU Soft Reboot for DT
478 [Documentation] Test soft reboot of the ONU using voltctl command
479 [Tags] VOL-2820 ONUSoftRebootDt notready
480 [Setup] Start Logging ONUSoftRebootDt
481 #... AND Setup
482 [Teardown] Run Keywords Collect Logs
483 ... AND Stop Logging ONUSoftRebootDt
484 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530485 #Reboot the ONU and verify that ping fails
486 FOR ${I} IN RANGE 0 ${num_onus}
487 ${src}= Set Variable ${hosts.src[${I}]}
488 ${dst}= Set Variable ${hosts.dst[${I}]}
489 ${onu_device_id}= Get Device ID From SN ${src['onu']}
490 ${onu_port}= Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
491 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
492 Reboot Device ${onu_device_id}
493 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
494 ... Wait Until Keyword Succeeds 60s 2s
495 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
496 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
497 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
498 # Remove Subscriber Access (To replicate DT workflow)
499 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${ONOS_SSH_IP}
500 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
501 # Delete ONU Device (To replicate DT workflow)
502 Delete Device ${onu_device_id}
503 Sleep 40s
504 # Check ONU port is Enabled in ONOS
505 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Suchitra Vemuri760bdd32020-06-17 12:34:48 -0700506 ... Verify ONU Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hema8c1a0322020-05-26 16:02:32 +0530507 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
508 ... Execute ONOS CLI Command ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
509 ... volt-add-subscriber-access ${of_id} ${onu_port}
510 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700511 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530512 ... ENABLED ACTIVE REACHABLE
513 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
514 # Verify subscriber access flows are added for the ONU port
515 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
516 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
517 ... ${onu_port} ${nni_port} ${src['s_tag']}
518 #Run Keyword If ${has_dataplane} Clean Up Linux
519 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
520 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
521 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
522 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
523 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530524 END
525 # Verify ONOS Flows
526 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
527 ${onos_flows_count}= Evaluate 4 * ${num_onus}
528 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
529 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
530 ... ${of_id} ${onos_flows_count}
531 # Verify VOLTHA Flows
532 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
533 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
534 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
535 ${onu_flows}= Set Variable 2
536 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
537 ${List_ONU_Serial} Create List
538 Set Suite Variable ${List_ONU_Serial}
539 Build ONU SN List ${List_ONU_Serial}
540 Log ${List_ONU_Serial}
541 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
542 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530543
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530544*** Keywords ***
545Setup Suite
546 [Documentation] Set up the test suite
547 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200548 #power_switch.robot needs it to support different vendor's power switch
549 ${switch_type}= Get Variable Value ${web_power_switch.type}
550 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000551 # Run Pre-test Setup for Soak Job
552 # Note: As soak requirement, it expects that the devices under test are already created and enabled
553 Run Keyword If '${SOAK_TEST}'=='True' Setup Soak
554
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530555
556Clear All Devices Then Create New Device
557 [Documentation] Remove any devices from VOLTHA and ONOS
558 # Remove all devices from voltha and nos
559 Delete All Devices and Verify
560 # Execute normal test Setup Keyword
561 Setup
562