blob: 3888ccdf28f0a8ecfd34138f8120dab8f0b62d62 [file] [log] [blame]
Hardik Windlassb9bdd502020-04-17 14:46:21 +05301# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test various end-to-end scenarios
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
Andrea Campanella15619c72020-04-30 09:48:04 +020034Resource ../../libraries/power_switch.robot
Hardik Windlassb9bdd502020-04-17 14:46:21 +053035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Matteo Scandoloa4e59522021-08-31 14:56:24 -070046${STACK_NAME} voltha
Hardik Windlassb9bdd502020-04-17 14:46:21 +053047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
53${has_dataplane} True
54${teardown_device} False
55${scripts} ../../scripts
56
57# Per-test logging on failure is turned off by default; set this variable to enable
58${container_log_dir} ${None}
59
Hardik Windlass43531332021-05-19 07:07:03 +000060# logging flag to enable Collect Logs, can be passed via the command line too
61# example: -v logging:False
62${logging} True
63
Hardik Windlass4ed54502021-05-14 07:34:15 +000064# Flag specific to Soak Jobs
65${SOAK_TEST} False
66
Hardik Windlassb9bdd502020-04-17 14:46:21 +053067*** Test Cases ***
68Verify ONU after Rebooting Physically for DT
69 [Documentation] Test the ONU functionality by physically turning on/off ONU.
70 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
71 ... Test case runs only on the PODs that are configured with PowerSwitch that
72 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
73 [Tags] functionalDt PowerSwitchOnuRebootDt VOL-2819 PowerSwitch
Hardik Windlass9df139e2020-04-24 14:54:54 +053074 [Setup] Start Logging RebootOnu_PowerSwitch_Dt
Hardik Windlass43531332021-05-19 07:07:03 +000075 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070077 ... AND Run Keyword If ${logging} Collect Logs
78 ... AND Stop Logging RebootOnu_PowerSwitch_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +053079 # Add OLT device
80 Setup
81 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
82 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +000083 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +053084 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Hardik Windlass2eb47182020-05-19 20:24:12 +053085 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080086 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +053087 ${src}= Set Variable ${hosts.src[${I}]}
88 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080089 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +000090 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass0560f802020-05-06 23:06:53 +053091 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +000092 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +053093 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass7c801e62020-05-04 19:33:21 +053094 # Disable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +053095 Disable Switch Outlet ${src['power_switch_port']}
96 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +000097 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +053098 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
99 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530100 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000101 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Girish Gowdrae0c711e2021-07-20 14:30:17 -0700103 # Allow the remove subscriber command to clean up all the flows, schedulers and queues before deleting the device
Girish Gowdrab6830612021-10-18 17:11:23 -0700104 Sleep 30s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530105 # Delete ONU Device (To replicate DT workflow)
106 Delete Device ${onu_device_id}
107 Sleep 5s
108 # Enable Power Switch
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530109 Enable Switch Outlet ${src['power_switch_port']}
110 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000111 Wait Until Keyword Succeeds 120s 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000112 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000113 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000114 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530115 ... volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530116 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000117 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530118 ... ENABLED ACTIVE REACHABLE
Hardik Windlass7c801e62020-05-04 19:33:21 +0530119 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530120 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000121 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530122 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
123 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass2eb47182020-05-19 20:24:12 +0530124 #Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass7c801e62020-05-04 19:33:21 +0530125 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
126 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530127 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530128 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
129 ... ${dst['container_name']}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530130 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800131 # Verify flows for all OLTs
132 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate All OLT Flows
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530133
134Verify OLT after Rebooting Physically for DT
135 [Documentation] Test the physical reboot of the OLT
136 ... Assuming that all the ONUs are DHCP/pingable (i.e. assuming sanityDt test was executed)
137 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
138 [Tags] functionalDt PhysicalOltRebootDt VOL-2817
Hardik Windlass9df139e2020-04-24 14:54:54 +0530139 [Setup] Start Logging RebootOlt_Physical_Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000140 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800141 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700142 ... AND Run Keyword If ${logging} Collect Logs
143 ... AND Stop Logging RebootOlt_Physical_Dt
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530144 # Add OLT device
145 Setup
146 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
147 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000148 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530149 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800150 FOR ${I} IN RANGE 0 ${olt_count}
151 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
152 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
153 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
154 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
155 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
156 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100157 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800158 END
159 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530160 ${src}= Set Variable ${hosts.src[${I}]}
161 ${dst}= Set Variable ${hosts.dst[${I}]}
162 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000163 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530164 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
165 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
166 END
167 # Wait for the OLT to come back up
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800168 FOR ${I} IN RANGE 0 ${olt_count}
169 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
170 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
171 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
172 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
173 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
174 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
175 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000176 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800177 ... Validate OLT Device ENABLED ACTIVE
178 ... REACHABLE ${olt_serial_number}
179 END
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530180 # Waiting extra time for the ONUs to come up
181 Sleep 60s
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530182 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000183 Perform Sanity Test DT
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530184
TorstenThiemeed3332f2021-02-02 10:30:48 +0000185Verify restart openonu-adapter container after subscriber provisioning for DT
186 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
187 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000188 [Tags] Restart-OpenOnu-Dt soak
TorstenThiemeed3332f2021-02-02 10:30:48 +0000189 [Setup] Start Logging Restart-OpenOnu-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000190 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
TorstenThiemeed3332f2021-02-02 10:30:48 +0000191 ... AND Stop Logging Restart-OpenOnu-Dt
TorstenThiemeed3332f2021-02-02 10:30:48 +0000192 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000193 Run Keyword If '${SOAK_TEST}'=='False' Setup
TorstenThiemeed3332f2021-02-02 10:30:48 +0000194 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
195 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000196 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000197 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
198 Log ${podStatusOutput}
199 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
200 ${podName} Set Variable adapter-open-onu
201 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
TorstenThiemeed3332f2021-02-02 10:30:48 +0000202 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
203 ... app ${podName} Running
204 # Wait for 1 min after openonu adapter is restarted
205 Sleep 60s
206 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000207 Perform Sanity Test DT
TorstenThiemeed3332f2021-02-02 10:30:48 +0000208 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
209 Log ${podStatusOutput}
210 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
211 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
212 Log to console Pod ${podName} restarted and sanity checks passed successfully
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000213 Run Keyword If '${SOAK_TEST}'=='False' Delete All Devices and Verify
TorstenThiemeed3332f2021-02-02 10:30:48 +0000214
Hardik Windlass9df139e2020-04-24 14:54:54 +0530215Verify restart openolt-adapter container after subscriber provisioning for DT
216 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
217 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000218 [Tags] Restart-OpenOlt-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530219 [Setup] Start Logging Restart-OpenOlt-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000220 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530221 ... AND Stop Logging Restart-OpenOlt-Dt
222 # Add OLT device
Hardik Windlass3280f2d2021-05-20 07:38:31 +0000223 Run Keyword If '${SOAK_TEST}'=='False' setup
Hardik Windlass9df139e2020-04-24 14:54:54 +0530224 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
225 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000226 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530227 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
228 Log ${podStatusOutput}
229 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100230 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530231 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530232 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
233 ... app ${podName} Running
TorstenThiemeed3332f2021-02-02 10:30:48 +0000234 # Wait for 1 min after openolt adapter is restarted
Hardik Windlass9df139e2020-04-24 14:54:54 +0530235 Sleep 60s
236 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000237 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530238 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
239 Log ${podStatusOutput}
240 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
241 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
242 Log to console Pod ${podName} restarted and sanity checks passed successfully
243
244Verify openolt adapter restart before subscriber provisioning for DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000245 [Documentation] Restart openolt-adapter container before adding the subscriber.
246 [Tags] functionalDt olt-adapter-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530247 [Setup] Start Logging OltAdapterRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000248 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530249 ... AND Stop Logging OltAdapterRestart-Dt
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000250 # Add OLT device
251 Clear All Devices Then Create New Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530252 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass9df139e2020-04-24 14:54:54 +0530253 Set Global Variable ${of_id}
254
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800255 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530256 ${src}= Set Variable ${hosts.src[${I}]}
257 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800258 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530259 ${onu_device_id}= Get Device ID From SN ${src['onu']}
260 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
261 ... ${of_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000262 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000263 ... Validate Device ENABLED ACTIVE REACHABLE
264 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530265 END
266 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100267 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
268 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
269 ... ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530270 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100271 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530272 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100273 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530274
275 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800276 FOR ${I} IN RANGE 0 ${olt_count}
277 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
278 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
279 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
280 ... ${olt_serial_number}
281 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
282 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
283 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530284
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800285 FOR ${I} IN RANGE 0 ${num_all_onus}
286 ${src}= Set Variable ${hosts.src[${I}]}
287 ${dst}= Set Variable ${hosts.dst[${I}]}
288 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000289 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800290 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
291 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530292 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000293 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530294 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
295 # Verify subscriber access flows are added for the ONU port
296 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
297 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
298 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000299 Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000300 ... ENABLED ACTIVE REACHABLE
301 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass9df139e2020-04-24 14:54:54 +0530302 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
303 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
304 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
305 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
306 ... ${dst['container_name']}
307 END
308
309Verify restart ofagent container after subscriber is provisioned for DT
310 [Documentation] Restart ofagent container after VOLTHA is operational.
311 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass4ed54502021-05-14 07:34:15 +0000312 [Tags] functionalDt ofagentRestart-Dt soak
Hardik Windlass9df139e2020-04-24 14:54:54 +0530313 [Setup] Start Logging ofagentRestart-Dt
Hardik Windlass43531332021-05-19 07:07:03 +0000314 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530315 ... AND Stop Logging ofagentRestart-Dt
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700316 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530317 # set timeout value
318 ${waitforRestart} Set Variable 120s
319 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
320 Log ${podStatusOutput}
321 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
322 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530323 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530324 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
325 ... app ${podName} Running
Hardik Windlass5e214b22021-02-26 10:37:14 +0000326 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530327 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
328 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000329 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530330 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
331 Log ${podStatusOutput}
332 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
333 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
334 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700335 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530336 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800337 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530338 ${src}= Set Variable ${hosts.src[${I}]}
339 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800340 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000341 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530342 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000343 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530344 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
345 # Verify ONU state in voltha
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000346 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s Validate Device
Hardik Windlass9df139e2020-04-24 14:54:54 +0530347 ... ENABLED ACTIVE REACHABLE
348 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
349 # Check ONU port is Disabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000350 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass39015672021-07-05 05:48:08 +0000351 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530352 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000353 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530354 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
355 ... ${onu_port} ${nni_port} ${src['s_tag']}
356 # Verify Ping
357 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
358 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
359 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530360 END
361 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700362 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530363 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
364 ... Running
365 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
366 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000367 Perform Sanity Test DT
Hardik Windlass9df139e2020-04-24 14:54:54 +0530368 Log to console Pod ${podName} restarted and sanity checks passed successfully
369
370Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart for DT
371 [Documentation] Deploys an device instance and waits for it to authenticate. After
372 ... authentication is successful the rw-core deployment is scaled to 0 instances to
373 ... simulate a POD crash. The test then scales the rw-core back to a single instance
374 ... and configures ONOS for access. The test succeeds if the device is able to
375 ... complete the DHCP sequence.
Hardik Windlass43531332021-05-19 07:07:03 +0000376 [Tags] functionalDt rwcore-restart-Dt
Hardik Windlass9df139e2020-04-24 14:54:54 +0530377 [Setup] Run Keywords Start Logging RwCoreFailAndRestart-Dt
378 ... AND Clear All Devices Then Create New Device
Hardik Windlass43531332021-05-19 07:07:03 +0000379 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hardik Windlass9df139e2020-04-24 14:54:54 +0530380 ... AND Stop Logging RwCoreFailAndRestart-Dt
381 #... AND Delete Device and Verify
Hardik Windlass9df139e2020-04-24 14:54:54 +0530382 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800383 FOR ${I} IN RANGE 0 ${olt_count}
384 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
385 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
386 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
387 ... ${olt_serial_number}
TorstenThiemedda37652021-11-11 16:22:05 +0000388 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800389 END
390 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530391 ${src}= Set Variable ${hosts.src[${I}]}
392 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800393 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530394 ${onu_device_id}= Get Device ID From SN ${src['onu']}
395 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
396 ... ${of_id}
397 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700398 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000399 ... ${onu_device_id} onu=True onu_reason=initial-mib-downloaded by_dev_id=True
Hardik Windlass9df139e2020-04-24 14:54:54 +0530400 END
401
402 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530403 Scale K8s Deployment voltha voltha-voltha-rw-core 0
404 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-voltha-rw-core
Hardik Windlass9df139e2020-04-24 14:54:54 +0530405 # Ensure the ofagent POD goes "not-ready" as expected
406 Wait Until keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530407 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 0
Hardik Windlass9df139e2020-04-24 14:54:54 +0530408 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530409 Scale K8s Deployment voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530410 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530411 ... Check Expected Available Deployment Replicas voltha voltha-voltha-rw-core 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530412 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530413 ... Check Expected Available Deployment Replicas voltha voltha-voltha-ofagent 1
Hardik Windlass9df139e2020-04-24 14:54:54 +0530414 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
415 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700416 Restart VOLTHA Port Forward voltha-api
Hardik Windlass9df139e2020-04-24 14:54:54 +0530417 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
418 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800419 FOR ${I} IN RANGE 0 ${olt_count}
420 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
421 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
422 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
423 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700424 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800425 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
426 END
Hardik Windlass9df139e2020-04-24 14:54:54 +0530427
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800428 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700429 ${src}= Set Variable ${hosts.src[${I}]}
430 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800431 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000432 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Andy Bavier52094622020-05-12 15:54:24 -0700433 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
434 ... ${of_id}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530435 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000436 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass9df139e2020-04-24 14:54:54 +0530437 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
438 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000439 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass9df139e2020-04-24 14:54:54 +0530440 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
441 ... ${onu_port} ${nni_port} ${src['s_tag']}
442 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
443 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
444 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
445 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
446 ... ${dst['container_name']}
447 END
448
Hemadf003682020-04-28 21:22:22 +0530449Verify OLT Soft Reboot for DT
450 [Documentation] Test soft reboot of the OLT using voltctl command
Hardik Windlass20a503d2021-06-21 11:58:08 +0000451 [Tags] VOL-2818 OLTSoftRebootDt functionalDt
Hemadf003682020-04-28 21:22:22 +0530452 [Setup] Start Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000453 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
Hemadf003682020-04-28 21:22:22 +0530454 ... AND Stop Logging OLTSoftRebootDt
Hardik Windlass20a503d2021-06-21 11:58:08 +0000455 FOR ${I} IN RANGE 0 ${olt_count}
456 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
457 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
458 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
459 ... Validate OLT Device ENABLED ACTIVE
460 ... REACHABLE ${olt_serial_number}
461 # Reboot the OLT using "voltctl device reboot" command
462 Reboot Device ${olt_device_id}
463 # Wait for the OLT to actually go down
464 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
465 ... ${olt_serial_number}
466 END
Hemadf003682020-04-28 21:22:22 +0530467 #Verify that ping fails
Hardik Windlass20a503d2021-06-21 11:58:08 +0000468 FOR ${I} IN RANGE 0 ${num_all_onus}
Hemadf003682020-04-28 21:22:22 +0530469 ${src}= Set Variable ${hosts.src[${I}]}
470 ${dst}= Set Variable ${hosts.dst[${I}]}
471 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000472 ... Wait Until Keyword Succeeds ${timeout} 2s
Hemadf003682020-04-28 21:22:22 +0530473 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
474 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
475 END
Hardik Windlass20a503d2021-06-21 11:58:08 +0000476 # Check OLT states
477 FOR ${I} IN RANGE 0 ${olt_count}
478 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
479 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
480 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
481 # Wait for the OLT to come back up
482 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
483 ... Check Remote System Reachability True ${olt_ssh_ip}
484 # Check OLT states
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000485 Wait Until Keyword Succeeds 360s 5s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000486 ... Validate OLT Device ENABLED ACTIVE
487 ... REACHABLE ${olt_serial_number}
488 END
Hemadf003682020-04-28 21:22:22 +0530489 # Waiting extra time for the ONUs to come up
490 Sleep 60s
Hardik Windlass20a503d2021-06-21 11:58:08 +0000491 #Check after reboot that ONUs are active, DHCP/pingable
Hemadf003682020-04-28 21:22:22 +0530492 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000493 Perform Sanity Test DT
Hemadf003682020-04-28 21:22:22 +0530494
Hema8c1a0322020-05-26 16:02:32 +0530495Verify ONU Soft Reboot for DT
496 [Documentation] Test soft reboot of the ONU using voltctl command
497 [Tags] VOL-2820 ONUSoftRebootDt notready
498 [Setup] Start Logging ONUSoftRebootDt
499 #... AND Setup
500 [Teardown] Run Keywords Collect Logs
501 ... AND Stop Logging ONUSoftRebootDt
502 #... AND Delete Device and Verify
Hema8c1a0322020-05-26 16:02:32 +0530503 #Reboot the ONU and verify that ping fails
504 FOR ${I} IN RANGE 0 ${num_onus}
505 ${src}= Set Variable ${hosts.src[${I}]}
506 ${dst}= Set Variable ${hosts.dst[${I}]}
TorstenThiemedda37652021-11-11 16:22:05 +0000507 ${of_id}= Get ofID From OLT List ${src['olt']}
508 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hema8c1a0322020-05-26 16:02:32 +0530509 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000510 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hema8c1a0322020-05-26 16:02:32 +0530511 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
512 Reboot Device ${onu_device_id}
513 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000514 ... Wait Until Keyword Succeeds ${timeout} 2s
Hema8c1a0322020-05-26 16:02:32 +0530515 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
516 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
517 Verify ping is succesful except for given device ${num_onus} ${onu_device_id}
518 # Remove Subscriber Access (To replicate DT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000519 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hema8c1a0322020-05-26 16:02:32 +0530520 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
521 # Delete ONU Device (To replicate DT workflow)
522 Delete Device ${onu_device_id}
523 Sleep 40s
524 # Check ONU port is Enabled in ONOS
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000525 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000526 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000527 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000528 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hema8c1a0322020-05-26 16:02:32 +0530529 ... volt-add-subscriber-access ${of_id} ${onu_port}
530 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700531 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hema8c1a0322020-05-26 16:02:32 +0530532 ... ENABLED ACTIVE REACHABLE
533 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
534 # Verify subscriber access flows are added for the ONU port
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000535 Wait Until Keyword Succeeds ${timeout} 5s
Hema8c1a0322020-05-26 16:02:32 +0530536 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
537 ... ${onu_port} ${nni_port} ${src['s_tag']}
538 #Run Keyword If ${has_dataplane} Clean Up Linux
539 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
540 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
541 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
542 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
543 ... ${dst['container_name']}
Hema8c1a0322020-05-26 16:02:32 +0530544 END
545 # Verify ONOS Flows
546 # Number of Access Flows on ONOS equals 4 * the Number of Active ONUs (2 for each downstream and upstream)
547 ${onos_flows_count}= Evaluate 4 * ${num_onus}
548 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
549 ... Verify Subscriber Access Flows Added Count DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
550 ... ${of_id} ${onos_flows_count}
551 # Verify VOLTHA Flows
552 # Number of per OLT Flows equals Twice the Number of Active ONUs (each for downstream and upstream) + 1 for LLDP
553 ${olt_flows}= Evaluate 2 * ${num_onus} + 1
554 # Number of per ONU Flows equals 2 (one each for downstream and upstream)
555 ${onu_flows}= Set Variable 2
556 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Flows ${olt_flows}
557 ${List_ONU_Serial} Create List
558 Set Suite Variable ${List_ONU_Serial}
559 Build ONU SN List ${List_ONU_Serial}
560 Log ${List_ONU_Serial}
561 Run Keyword Wait Until Keyword Succeeds ${timeout} 5s Validate ONU Flows
562 ... ${List_ONU_Serial} ${onu_flows}
Hemadf003682020-04-28 21:22:22 +0530563
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000564Verify restart openonu-adapter container for DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000565 [Documentation] Restart openonu-adapter container after VOLTHA is operational.
566 ... Run the ping continuously in background during container restart,
567 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000568 ... Also, verify that the voltha control plane functionality is not affected.
569 [Tags] functionalDt RestartOpenOnuPingDt
Hardik Windlassd19af862021-08-12 11:40:20 +0000570 [Setup] Start Logging RestartOpenOnuPingDt
571 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
572 ... AND Stop Logging RestartOpenOnuPingDt
573 Clear All Devices Then Create New Device
574 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
575 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000576 Perform Sanity Test DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000577 FOR ${I} IN RANGE 0 ${num_all_onus}
578 ${src}= Set Variable ${hosts.src[${I}]}
579 ${dst}= Set Variable ${hosts.dst[${I}]}
580 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
581 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000582 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000583 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
584 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
585 END
586 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
587 Log ${podStatusOutput}
588 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
589 ${podName} Set Variable adapter-open-onu
590 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
591 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
592 ... app ${podName} Running
593 # Wait for 1 min after openonu adapter is restarted
594 Sleep 60s
595 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
596 Log ${podStatusOutput}
597 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
598 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
599 FOR ${I} IN RANGE 0 ${num_all_onus}
600 ${src}= Set Variable ${hosts.src[${I}]}
601 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassd19af862021-08-12 11:40:20 +0000602 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000603 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000604 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
605 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000606 END
607 FOR ${I} IN RANGE 0 ${num_all_onus}
608 ${src}= Set Variable ${hosts.src[${I}]}
609 ${dst}= Set Variable ${hosts.dst[${I}]}
610 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassd19af862021-08-12 11:40:20 +0000611 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000612 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassd19af862021-08-12 11:40:20 +0000613 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
614 ... ${src['container_type']} ${src['container_name']}
615 Check Ping Result True ${ping_output}
616 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000617 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
618 Verify Control Plane After Pod Restart DT
Hardik Windlassd19af862021-08-12 11:40:20 +0000619
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000620Verify restart openolt-adapter container for DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000621 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
622 ... Run the ping continuously in background during container restart,
623 ... and verify that there should be no affect on the dataplane.
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000624 ... Also, verify that the voltha control plane functionality is not affected.
625 [Tags] functionalDt RestartOpenOltPingDt
Hardik Windlassa04efa52021-08-13 08:17:24 +0000626 [Setup] Start Logging RestartOpenOltPingDt
627 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
628 ... AND Stop Logging RestartOpenOltPingDt
629 Clear All Devices Then Create New Device
630 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
631 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000632 Perform Sanity Test DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000633 FOR ${I} IN RANGE 0 ${num_all_onus}
634 ${src}= Set Variable ${hosts.src[${I}]}
635 ${dst}= Set Variable ${hosts.dst[${I}]}
636 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
637 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000638 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000639 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
640 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
641 END
642 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
643 Log ${podStatusOutput}
644 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
645 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
646 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
647 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
648 ... app ${podName} Running
649 # Wait for 1 min after openolt adapter is restarted
650 Sleep 60s
651 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
652 Log ${podStatusOutput}
653 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
654 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
655 FOR ${I} IN RANGE 0 ${num_all_onus}
656 ${src}= Set Variable ${hosts.src[${I}]}
657 ${dst}= Set Variable ${hosts.dst[${I}]}
Hardik Windlassa04efa52021-08-13 08:17:24 +0000658 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000659 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000660 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
661 ... ${src['container_type']} ${src['container_name']}
Hardik Windlass6763ba52021-08-17 03:14:23 +0000662 END
663 FOR ${I} IN RANGE 0 ${num_all_onus}
664 ${src}= Set Variable ${hosts.src[${I}]}
665 ${dst}= Set Variable ${hosts.dst[${I}]}
666 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
Hardik Windlassa04efa52021-08-13 08:17:24 +0000667 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000668 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlassa04efa52021-08-13 08:17:24 +0000669 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
670 ... ${src['container_type']} ${src['container_name']}
671 Check Ping Result True ${ping_output}
672 END
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000673 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
674 Verify Control Plane After Pod Restart DT
675
676Verify restart rw-core container for DT
677 [Documentation] Restart rw-core container after VOLTHA is operational.
678 ... Run the ping continuously in background during container restart,
679 ... and verify that there should be no affect on the dataplane.
680 ... Also, verify that the voltha control plane functionality is not affected.
681 [Tags] functionalDt RestartRwCorePingDt
682 [Setup] Start Logging RestartRwCorePingDt
683 [Teardown] Run Keywords Run Keyword If ${logging} Collect Logs
684 ... AND Stop Logging RestartRwCorePingDt
685 Clear All Devices Then Create New Device
686 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
687 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000688 Perform Sanity Test DT
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000689 FOR ${I} IN RANGE 0 ${num_all_onus}
690 ${src}= Set Variable ${hosts.src[${I}]}
691 ${dst}= Set Variable ${hosts.dst[${I}]}
692 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
693 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000694 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000695 ... Run Ping In Background ${ping_output_file} ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
696 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
697 END
698 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
699 Log ${podStatusOutput}
700 ${countBeforeRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
701 ${podName} Set Variable rw-core
702 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
703 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
704 ... app ${podName} Running
705 # Wait for 1 min after rw-core is restarted
706 Sleep 60s
707 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
708 # so restart the port forwarding for the API service
709 Restart VOLTHA Port Forward voltha-api
710 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
711 Log ${podStatusOutput}
712 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
713 Should Be Equal As Strings ${countAfterRestart} ${countBeforeRestart}
714 FOR ${I} IN RANGE 0 ${num_all_onus}
715 ${src}= Set Variable ${hosts.src[${I}]}
716 ${dst}= Set Variable ${hosts.dst[${I}]}
717 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000718 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000719 ... Stop Ping Running In Background ${src['ip']} ${src['user']} ${src['pass']}
720 ... ${src['container_type']} ${src['container_name']}
721 END
722 FOR ${I} IN RANGE 0 ${num_all_onus}
723 ${src}= Set Variable ${hosts.src[${I}]}
724 ${dst}= Set Variable ${hosts.dst[${I}]}
725 ${ping_output_file}= Set Variable /tmp/${src['onu']}_ping
726 ${ping_output}= Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000727 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000728 ... Retrieve Remote File Contents ${ping_output_file} ${src['ip']} ${src['user']} ${src['pass']}
729 ... ${src['container_type']} ${src['container_name']}
730 Check Ping Result True ${ping_output}
731 END
732 # Verify Control Plane Functionality by Deleting and Re-adding the Subscriber
733 Verify Control Plane After Pod Restart DT
Hardik Windlassa04efa52021-08-13 08:17:24 +0000734
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530735*** Keywords ***
736Setup Suite
737 [Documentation] Set up the test suite
738 Common Test Suite Setup
Andrea Campanella15619c72020-04-30 09:48:04 +0200739 #power_switch.robot needs it to support different vendor's power switch
740 ${switch_type}= Get Variable Value ${web_power_switch.type}
741 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Hardik Windlass4ed54502021-05-14 07:34:15 +0000742 # Run Pre-test Setup for Soak Job
743 # Note: As soak requirement, it expects that the devices under test are already created and enabled
744 Run Keyword If '${SOAK_TEST}'=='True' Setup Soak
745
Hardik Windlassb9bdd502020-04-17 14:46:21 +0530746
747Clear All Devices Then Create New Device
748 [Documentation] Remove any devices from VOLTHA and ONOS
749 # Remove all devices from voltha and nos
750 Delete All Devices and Verify
751 # Execute normal test Setup Keyword
752 Setup
753
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000754Verify Control Plane After Pod Restart DT
755 [Documentation] Verifies the control plane functionality after the voltha pod restart
756 ... by deleting and re-adding the subscriber
757 FOR ${I} IN RANGE 0 ${num_all_onus}
758 ${src}= Set Variable ${hosts.src[${I}]}
759 ${dst}= Set Variable ${hosts.dst[${I}]}
760 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000761 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000762 ${onu_device_id}= Get Device ID From SN ${src['onu']}
763 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
764 ... ${of_id}
765 # Remove Subscriber Access
766 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
767 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
768 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000769 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000770 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
771 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
772 # Disable and Re-Enable the ONU (To replicate DT current workflow)
773 # TODO: Delete and Auto-Discovery Add of ONU (not yet supported)
774 Disable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000775 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000776 ... Validate Device DISABLED UNKNOWN
777 ... REACHABLE ${src['onu']}
778 Enable Device ${onu_device_id}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000779 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000780 ... Validate Device ENABLED ACTIVE
781 ... REACHABLE ${src['onu']}
782 # Add Subscriber Access
783 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
784 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
785 # Verify subscriber access flows are added for the ONU port
786 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
787 ... Verify Subscriber Access Flows Added For ONU DT ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
788 ... ${onu_port} ${nni_port} ${src['s_tag']}
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000789 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000790 ... Validate Device ENABLED ACTIVE
791 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
792 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass5d537ad2021-11-16 06:01:54 +0000793 ... Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlasscf46ec42021-09-28 08:05:49 +0000794 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
795 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
796 END