blob: 6901d4dc8d88a97cda127f78d8cb0d48db252a6f [file] [log] [blame]
Gayathri.Selvan283a63c2020-01-23 04:09:18 +00001# Copyright 2017 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000014
15*** Settings ***
HungWei Chiu87edb1f2020-01-28 00:10:58 +000016Documentation Test various failure scenarios
HungWei Chiu1408fb92020-03-03 19:43:30 -050017Suite Setup Setup Suite
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000018Test Setup Setup
19Test Teardown Teardown
TorstenThieme4e2168e2021-06-22 14:01:47 +000020Suite Teardown Teardown Suite
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000021Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
27Resource ../../libraries/onos.robot
28Resource ../../libraries/voltctl.robot
29Resource ../../libraries/voltha.robot
30Resource ../../libraries/utils.robot
31Resource ../../libraries/k8s.robot
32Resource ../../variables/variables.robot
HungWei Chiu87edb1f2020-01-28 00:10:58 +000033Resource ../../libraries/power_switch.robot
Hardik Windlassc082c422021-03-16 15:44:10 +000034Resource ../../libraries/bbsim.robot
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000035
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Matteo Scandoloa4e59522021-08-31 14:56:24 -070045${STACK_NAME} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000046${INFRA_NAMESPACE} default
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000047# For below variable value, using deployment name as using grep for
48# parsing radius pod name, we can also use full radius pod name
49${RESTART_POD_NAME} radius
50${timeout} 60s
51${of_id} 0
52${logical_id} 0
Suchitra Vemuri109feb12020-01-23 16:25:02 -080053${has_dataplane} True
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000054${teardown_device} False
55${scripts} ../../scripts
56
Matteo Scandolo7e519fb2021-08-13 11:35:16 -070057${suppressaddsubscriber} True
58
Andy Bavierabeba262020-02-07 16:22:16 -070059# Per-test logging on failure is turned off by default; set this variable to enable
60${container_log_dir} ${None}
61
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070062# logging flag to enable Collect Logs, can be passed via the command line too
63# example: -v logging:False
64${logging} True
65
Gayathri.Selvan283a63c2020-01-23 04:09:18 +000066*** Test Cases ***
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080067Verify ONU after rebooting physically
68 [Documentation] Test the ONU functionality by physically turning on/off ONU.
69 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
70 ... Test case runs only on the PODs that are configured with PowerSwitch that
71 ... controls the power off/on ONUs/OLT remotely (simulating a physical reboot)
72 ... VOL-2634
73 [Tags] functional PowerSwitch
Andy Baviere187eda2020-04-20 15:00:02 -070074 [Setup] Start Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080075 [Teardown] Run Keywords Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080076 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -070077 ... AND Run Keyword If ${logging} Collect Logs
78 ... AND Stop Logging ONUreboot_PowerSwitch
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080079 # Add OLT device
80 setup
81 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
82 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +053083 Perform Sanity Test
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080084 Power Switch Connection Suite ${web_power_switch.ip} ${web_power_switch.user} ${web_power_switch.password}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080085 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080086 ${src}= Set Variable ${hosts.src[${I}]}
87 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -080088 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +000089 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass58b23a32022-05-10 11:06:48 +053090 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass1ed2eee2021-06-25 09:51:03 +000091 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +053092 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Girish Gowdraa3a05c62022-02-16 13:54:37 -080093 # If the power switch port is not specified, continue
94 Continue For Loop If '${src["power_switch_port"]}' == '${None}'
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080095 Disable Switch Outlet ${src['power_switch_port']}
96 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +053097 ... Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemuri60be98b2020-02-19 16:41:21 -080098 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
99 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530100 # Remove Subscriber Access (To replicate ATT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000101 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530102 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800103
104 Enable Switch Outlet ${src['power_switch_port']}
105 # Check ONU port is Enabled in ONOS
Hardik Windlass58b23a32022-05-10 11:06:48 +0530106 Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000107 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800108 # Verify EAPOL flows are added for the ONU port
109 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700110 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800111 # Verify ONU state in voltha
Hardik Windlass58b23a32022-05-10 11:06:48 +0530112 Wait Until Keyword Succeeds 360s 5s Validate Device
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800113 ... ENABLED ACTIVE REACHABLE
114 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hardik Windlass7c801e62020-05-04 19:33:21 +0530115 Run Keyword If ${has_dataplane} Clean Up Linux
116 # Perform Authentication
117 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
118 ... /tmp/wpa ${src['dp_iface_name']} log
119 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
120 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
121 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530122 Wait Until Keyword Succeeds ${timeout} 2
Hardik Windlass7c801e62020-05-04 19:33:21 +0530123 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530124 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000125 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530126 ... volt-add-subscriber-access ${of_id} ${onu_port}
127 # Verify that no pending flows exist for the ONU port
128 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
129 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530130 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530131 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530132 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
133 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530134 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
135 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800136 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass7c801e62020-05-04 19:33:21 +0530137 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
138 ... ${dst['container_name']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530139 Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass7c801e62020-05-04 19:33:21 +0530140 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800141 END
Suchitra Vemurid7096132020-02-21 14:50:47 -0800142 # Deleting OLT after tests completes independently (as this test doesn't not run on each POD)
Suchitra Vemuri0656bc22020-03-21 19:07:49 -0700143 #Run Keyword If ${has_dataplane} Delete Device and Verify
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800144
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700145Verify OLT after rebooting physically
146 [Documentation] Test the physical reboot of the OLT
147 ... Prerequisite : Subscriber are authenticated/DHCP/pingable state
148 ... Test performs a physical reboot, performs "reboot" from the OLT CLI
149 ... VOL-1956
150 [Tags] functional PhysicalOLTReboot
Andy Baviere187eda2020-04-20 15:00:02 -0700151 [Setup] Start Logging PhysicalOLTReboot
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700152 [Teardown] Run Keywords Collect Logs
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800153 ... AND Delete All Devices and Verify
Girish Gowdraf244d5b2021-09-01 17:16:16 -0700154 ... AND Run Keyword If ${logging} Collect Logs
155 ... AND Stop Logging PhysicalOLTReboot
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700156 # Add OLT device
157 setup
158 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
159 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530160 Perform Sanity Test
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700161 # Reboot the OLT from the OLT CLI
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800162 FOR ${I} IN RANGE 0 ${olt_count}
163 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
164 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
165 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
166 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
167 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
168 Run Keyword If ${has_dataplane} Login And Run Command On Remote System
Andrea Campanella5ef88142021-02-02 14:34:50 +0100169 ... reboot ${olt_ssh_ip} ${olt_user} ${olt_pass} prompt=#
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800170 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800171 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700172 ${src}= Set Variable ${hosts.src[${I}]}
173 ${dst}= Set Variable ${hosts.dst[${I}]}
174 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +0530175 ... Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700176 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
177 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
178 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800179 # Wait for the OLTs to come back up
180 FOR ${I} IN RANGE 0 ${olt_count}
181 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
182 ${olt_user}= Get From Dictionary ${list_olts}[${I}] user
183 ${olt_pass}= Get From Dictionary ${list_olts}[${I}] pass
184 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
185 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
186 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
187 ... Check Remote System Reachability True ${olt_ssh_ip}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530188 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800189 ... Validate OLT Device ENABLED ACTIVE
190 ... REACHABLE ${olt_serial_number}
191 END
Suchitra Vemuri64db9652020-03-22 21:11:13 -0700192 # Waiting extra time for the ONUs to come up
193 Sleep 60s
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700194 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530195 Perform Sanity Test
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700196 # Deleting OLT after test completes
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800197 #Run Keyword If ${has_dataplane} Delete All Devices and Verify
Suchitra Vemuri784e1ae2020-03-16 18:40:25 -0700198
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700199Verify restart openolt-adapter container after subscriber provisioning
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800200 [Documentation] Restart openolt-adapter container after VOLTHA is operational.
201 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800202 [Tags] functional VOL-1958 Restart-OpenOlt released
Andy Baviere187eda2020-04-20 15:00:02 -0700203 [Setup] Start Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800204 [Teardown] Run Keywords Collect Logs
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800205 ... AND Stop Logging Restart-OpenOlt
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800206 # Add OLT device
207 setup
208 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
209 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530210 Perform Sanity Test
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800211 ${waitforRestart} Set Variable 120s
212 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
213 Log ${podStatusOutput}
214 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Andrea Campanella3dcce272021-01-15 16:04:47 +0100215 ${podName} Set Variable ${OLT_ADAPTER_APP_LABEL}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530216 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530217 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
218 ... app ${podName} Running
Suchitra Vemuri312bfe02020-03-16 16:11:34 -0700219 # Wait for 1min after openolt adapter is restarted
Hardik Windlass5e214b22021-02-26 10:37:14 +0000220 # TBD: Need for this Sleep
Suchitra Vemuri312bfe02020-03-16 16:11:34 -0700221 Sleep 60s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800222 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530223 Perform Sanity Test ${suppressaddsubscriber}
Suchitra Vemuria7dae322020-02-19 22:38:18 -0800224 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
225 Log ${podStatusOutput}
226 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
227 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
228 Log to console Pod ${podName} restarted and sanity checks passed successfully
Suchitra Vemuri60be98b2020-02-19 16:41:21 -0800229
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800230Check OLT/ONU Authentication After Radius Pod Restart
231 [Documentation] After radius restart, triggers reassociation, checks status and
232 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
233 ... wpa supplicant is running in background hence it is recommended to remove
234 ... teardown from previous test or uncomment 'Teardown None'.
235 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
236 [Tags] functional RadiusRestart released
Andy Baviere187eda2020-04-20 15:00:02 -0700237 [Setup] Start Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800238 [Teardown] Run Keywords Collect Logs
239 ... AND Stop Logging RadiusRestart
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800240 ${waitforRestart} Set Variable 120s
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530241 ${podName} Set Variable radius
Matteo Scandolo10acf562021-05-03 15:35:16 -0700242 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${INFRA_NAMESPACE} app ${podName}
243 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pods Status By Label ${INFRA_NAMESPACE}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530244 ... app ${podName} Running
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800245 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800246 ${src}= Set Variable ${hosts.src[${I}]}
247 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800248 ${of_id}= Get ofID From OLT List ${src['olt']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800249 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000250 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530251 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800252 Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700253 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
254 ... ${src['c_tag']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800255 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
256 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
257 ... ${src['ip']} ${src['user']} ${src['pass']}
258 ... ${src['container_type']} ${src['container_name']}
ubuntu6b6e7d42020-03-02 12:35:42 -0800259 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800260 ... ${ONOS_SSH_PORT} ${onu_port}
261 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
262 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
263 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
264 ... ${src['ip']} ${src['user']} ${src['pass']}
265 ... ${src['container_type']} ${src['container_name']}
266 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
267 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
268 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
ubuntu6b6e7d42020-03-02 12:35:42 -0800269 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800270 END
271
Suchitra Vemuri05180cb2020-03-26 13:13:27 -0700272Verify openolt adapter restart before subscriber provisioning
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800273 [Documentation] Deploys an device instance and waits for it to authenticate. After
274 ... authentication is successful the rw-core deployment is scaled to 0 instances to
275 ... simulate a POD crash. The test then scales the rw-core back to a single instance
276 ... and configures ONOS for access. The test succeeds if the device is able to
277 ... complete the DHCP sequence.
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700278 [Tags] functional olt-adapter-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700279 [Setup] Start Logging OltAdapterRestart
Suchitra Vemuria0c086f2020-03-29 19:14:13 -0700280 #... AND Clear All Devices Then Create New Device
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800281 [Teardown] Run Keywords Collect Logs
282 ... AND Stop Logging OltAdapterRestart
Hung-Wei Chiud3576762020-03-25 18:06:26 -0700283 # Add OLT and perform sanity test
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700284 #setup
Suchitra Vemuri91537c82020-03-29 15:03:26 -0700285 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuri59ad2772020-03-26 22:27:27 -0700286 #Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800287 #Set Global Variable ${of_id}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800288
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800289 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800290 ${src}= Set Variable ${hosts.src[${I}]}
291 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800292 ${of_id}= Get ofID From OLT List ${src['olt']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800293 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000294 ... ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530295 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800296
297 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700298 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000299 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed by_dev_id=True
ubuntu6b6e7d42020-03-02 12:35:42 -0800300 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700301 ... ${ONOS_SSH_PORT} ${of_id} ${onu_port} ${src['c_tag']}
Andy Bavier84834d42020-02-25 13:49:50 -0700302 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
303 ... /tmp/wpa ${src['dp_iface_name']} log
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800304 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
305 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
Andy Bavier84834d42020-02-25 13:49:50 -0700306 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
ubuntu6b6e7d42020-03-02 12:35:42 -0800307 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800308 ... ${ONOS_SSH_PORT} ${onu_port}
309 END
310 # Scale down the open OLT adapter deployment to 0 PODs and once confirmed, scale it back to 1
Andrea Campanella3dcce272021-01-15 16:04:47 +0100311 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 0
312 Wait Until Keyword Succeeds ${timeout} 2s Pods Do Not Exist By Label ${NAMESPACE} app
313 ... ${OLT_ADAPTER_APP_LABEL}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800314 # Scale up the open OLT adapter deployment and make sure both it and the ofagent deployment are back
Andrea Campanella3dcce272021-01-15 16:04:47 +0100315 Scale K8s Deployment by Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800316 Wait Until Keyword Succeeds ${timeout} 2s
Andrea Campanella3dcce272021-01-15 16:04:47 +0100317 ... Check Expected Available Deployment Replicas By Pod Label ${NAMESPACE} app ${OLT_ADAPTER_APP_LABEL} 1
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800318
319 # Ensure the device is available in ONOS, this represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800320 FOR ${I} IN RANGE 0 ${olt_count}
321 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
322 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
323 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
324 ... ${olt_serial_number}
325 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
326 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
327 END
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800328
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800329 FOR ${I} IN RANGE 0 ${num_all_onus}
330 ${src}= Set Variable ${hosts.src[${I}]}
331 ${dst}= Set Variable ${hosts.dst[${I}]}
332 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000333 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800334 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000335 ... ${of_id} ${src['uni_id']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800336 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000337 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800338 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530339 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530340 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530341 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
342 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800343 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
344 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
345 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
346 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
347 ... ${dst['container_name']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530348 Wait Until Keyword Succeeds ${timeout} 2s
ubuntu6b6e7d42020-03-02 12:35:42 -0800349 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800350 END
351
hwchiu14c05102020-03-27 00:44:30 +0000352Verify restart ofagent container after subscriber is provisioned
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000353 [Documentation] Restart ofagent container after VOLTHA is operational.
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000354 ... Prerequisite : ONUs are authenticated and pingable.
Hardik Windlass7a846662020-03-19 20:42:59 +0530355 [Tags] functional VOL-2409 ofagentRestart
Andy Baviere187eda2020-04-20 15:00:02 -0700356 [Setup] Start Logging ofagentRestart
Andy Bavier4a8450e2020-02-04 08:58:37 -0700357 [Teardown] Run Keywords Collect Logs
Andy Bavierabeba262020-02-07 16:22:16 -0700358 ... AND Stop Logging ofagentRestart
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700359 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530360 ... AND Wait Until Keyword Succeeds ${timeout} 2s
361 ... Validate Pods Status By Label ${NAMESPACE} app ofagent Running
362 ... AND Wait Until Keyword Succeeds ${timeout} 3s
363 ... Pods Are Ready By Label ${NAMESPACE} app ofagent
Suchitra Vemuriceb002a2020-03-25 13:36:21 -0700364 # set timeout value
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000365 ${waitforRestart} Set Variable 120s
Andy Bavier4a8450e2020-02-04 08:58:37 -0700366 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000367 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700368 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530369 ${podName} Set Variable ofagent
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530370 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530371 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
372 ... app ${podName} Running
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530373 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
374 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530375 Perform Sanity Test ${suppressaddsubscriber}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700376 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000377 Log ${podStatusOutput}
Andy Bavier4a8450e2020-02-04 08:58:37 -0700378 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000379 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530380 # Scale Down the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700381 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530382 Sleep 30s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800383 FOR ${I} IN RANGE 0 ${num_all_onus}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530384 ${src}= Set Variable ${hosts.src[${I}]}
385 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800386 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000387 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530388 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000389 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530390 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530391 # Verify ONU state in voltha
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700392 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s Validate Device
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530393 ... ENABLED ACTIVE REACHABLE
394 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
395 # Check ONU port is Disabled in ONOS
Hardik Windlass58b23a32022-05-10 11:06:48 +0530396 Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000397 ... Verify UNI Port Is Disabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530398 # Verify EAPOL flows are present for the ONU port
399 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700400 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
401 ... ${src['c_tag']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530402 # Verify ONU in AAA-Users
403 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
404 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
405 # Verify DHCP-Allocations
406 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
407 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530408 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530409 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530410 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
411 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530412 # Verify Ping
413 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Check Ping True
414 ... ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']}
415 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530416 END
417 # Scale Up the Of-Agent Deployment
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700418 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530419 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
420 ... Running
Hardik Windlassf86db5f2022-04-14 10:18:00 +0530421 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${podName}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530422 # Performing Sanity Test to make sure subscribers are all AUTH+DHCP and pingable
423 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530424 Perform Sanity Test ${suppressaddsubscriber}
Hardik Windlass8f2aadc2020-03-16 20:16:39 +0530425 Log to console Pod ${podName} restarted and sanity checks passed successfully
Gayathri.Selvanb6a2b542020-01-24 07:24:23 +0000426
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000427Check ONU adapter crash not forcing authentication again
428 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
429 ... message count to make sure auth not started again and validates EAP status and ping.
430 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
hwchiu14c05102020-03-27 00:44:30 +0000431 [Tags] functional ONUAdaptCrash
Andy Baviere187eda2020-04-20 15:00:02 -0700432 [Setup] Start Logging ONUAdaptCrash
Andy Bavierabeba262020-02-07 16:22:16 -0700433 [Teardown] Run Keywords Collect Logs
434 ... AND Stop Logging ONUAdaptCrash
hwchiu14c05102020-03-27 00:44:30 +0000435 # Wait for adapter to resync
Hardik Windlass5e214b22021-02-26 10:37:14 +0000436 # TBD: Need for this Sleep
hwchiu14c05102020-03-27 00:44:30 +0000437 Sleep 60s
438 # Restart the onu
Hardik Windlass79b40ff2020-06-11 22:55:47 +0530439 ${podName} Set Variable adapter-open-onu
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700440 Wait Until Keyword Succeeds ${timeout} 15s Delete K8s Pods By Label ${NAMESPACE} app ${podName}
hwchiu14c05102020-03-27 00:44:30 +0000441 # Validate ONU Ports
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800442 FOR ${I} IN RANGE 0 ${num_all_onus}
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000443 ${src}= Set Variable ${hosts.src[${I}]}
444 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800445 ${of_id}= Get ofID From OLT List ${src['olt']}
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000446 ${onu_device_id}= Get Device ID From SN ${src['onu']}
447 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000448 ... ${of_id} ${src['uni_id']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530449 Wait Until Keyword Succeeds ${timeout} 2s Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000450 ... ${src['onu']} ${src['uni_id']}
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000451 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
452 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
453 ... ${src['container_type']} ${src['container_name']}
454 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Gayathri.Selvanc0b37022020-02-06 05:17:00 +0000455 END
Hung-Wei Chiu2bee4d42020-04-24 11:31:50 -0700456 Wait Until Keyword Succeeds ${timeout} 2s Validate Pods Status By Label ${NAMESPACE}
457 ... app ${podName} Running
hwchiu14c05102020-03-27 00:44:30 +0000458 # Wait for adapter to resync
Hardik Windlass5e214b22021-02-26 10:37:14 +0000459 # TBD: Need for this Sleep
hwchiu14c05102020-03-27 00:44:30 +0000460 Sleep 60s
Andy Bavier8da0e132020-04-13 10:25:16 -0700461 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800462 # Validate OLTs are active in ONOS
463 FOR ${I} IN RANGE 0 ${olt_count}
464 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
465 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
466 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
467 ... ${olt_serial_number}
468 END
hwchiu14c05102020-03-27 00:44:30 +0000469 # Perform all steps in Sanity Test except the subscriber addition
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800470 FOR ${I} IN RANGE 0 ${num_all_onus}
hwchiu14c05102020-03-27 00:44:30 +0000471 ${src}= Set Variable ${hosts.src[${I}]}
472 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800473 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000474 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
hwchiu14c05102020-03-27 00:44:30 +0000475 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530476 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000477 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
hwchiu14c05102020-03-27 00:44:30 +0000478 # Check ONU port is Enabled in ONOS
479 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000480 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
hwchiu14c05102020-03-27 00:44:30 +0000481 # Verify EAPOL flows are added for the ONU port
482 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700483 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
484 ... ${src['c_tag']}
hwchiu14c05102020-03-27 00:44:30 +0000485 # Verify ONU state in voltha
Hardik Windlass58b23a32022-05-10 11:06:48 +0530486 Wait Until Keyword Succeeds 360s 5s Validate Device
hwchiu14c05102020-03-27 00:44:30 +0000487 ... ENABLED ACTIVE REACHABLE
488 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
489 # Perform Authentication
490 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
491 ... /tmp/wpa ${src['dp_iface_name']} log
492 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
493 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
494 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
495 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2
496 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
497 # Verify that no pending flows exist for the ONU port
498 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
499 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530500 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530501 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530502 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
503 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
hwchiu14c05102020-03-27 00:44:30 +0000504 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
505 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
506 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
507 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
508 ... ${dst['container_name']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530509 Wait Until Keyword Succeeds ${timeout} 2s
hwchiu14c05102020-03-27 00:44:30 +0000510 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
hwchiu14c05102020-03-27 00:44:30 +0000511 END
512
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700513Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
514 [Documentation] Deploys an device instance and waits for it to authenticate. After
515 ... authentication is successful the rw-core deployment is scaled to 0 instances to
516 ... simulate a POD crash. The test then scales the rw-core back to a single instance
517 ... and configures ONOS for access. The test succeeds if the device is able to
518 ... complete the DHCP sequence.
519 [Tags] functional rwcore-restart
Andy Baviere187eda2020-04-20 15:00:02 -0700520 [Setup] Run Keywords Start Logging RwCoreFailAndRestart
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700521 ... AND Clear All Devices Then Create New Device
522 [Teardown] Run Keywords Collect Logs
523 ... AND Stop Logging RwCoreFailAndRestart
Suchitra Vemuri5d3383f2020-04-03 11:02:26 -0700524 #... AND Delete Device and Verify
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700525 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800526 FOR ${I} IN RANGE 0 ${olt_count}
527 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
528 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
529 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
530 ... ${olt_serial_number}
TorstenThiemedda37652021-11-11 16:22:05 +0000531 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800532 END
533 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700534 ${src}= Set Variable ${hosts.src[${I}]}
535 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800536 ${of_id}= Get ofID From OLT List ${src['olt']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700537 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000538 ... ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530539 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700540 # Bring up the device and verify it authenticates
Suchitra Vemuricdc32d82020-07-17 12:24:46 -0700541 Wait Until Keyword Succeeds 360s 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000542 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed by_dev_id=True
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700543 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700544 ... ${ONOS_SSH_PORT} ${of_id} ${onu_port}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700545 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
546 ... /tmp/wpa ${src['dp_iface_name']} log
547 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
548 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
549 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
550 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
551 ... ${ONOS_SSH_PORT} ${onu_port}
552 END
553
554 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700555 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-rw-core 0
556 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist ${NAMESPACE} ${STACK_NAME}-voltha-rw-core
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700557 # Ensure the ofagent POD goes "not-ready" as expected
558 Wait Until keyword Succeeds ${timeout} 2s
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700559 ... Check Expected Available Deployment Replicas ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 0
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700560 # Scale up the core deployment and make sure both it and the ofagent deployment are back
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700561 Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-rw-core 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700562 Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700563 ... Check Expected Available Deployment Replicas ${NAMESPACE} ${STACK_NAME}-voltha-rw-core 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700564 Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandoloa4e59522021-08-31 14:56:24 -0700565 ... Check Expected Available Deployment Replicas ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700566 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
567 # so restart the port forwarding for the API service
David Bainbridge07321ff2020-06-12 13:55:42 -0700568 Restart VOLTHA Port Forward voltha-api
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700569 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
570 # represents system connectivity being restored
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800571 FOR ${I} IN RANGE 0 ${olt_count}
572 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
573 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
574 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
575 ... ${olt_serial_number}
Girish Gowdrae4de3b32021-03-23 22:20:00 -0700576 Wait Until Keyword Succeeds 120s 2s Device Is Available In ONOS
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800577 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
578 END
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700579
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800580 FOR ${I} IN RANGE 0 ${num_all_onus}
Andy Bavier52094622020-05-12 15:54:24 -0700581 ${src}= Set Variable ${hosts.src[${I}]}
582 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800583 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000584 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Andy Bavier52094622020-05-12 15:54:24 -0700585 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000586 ... ${of_id} ${src['uni_id']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700587 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000588 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700589 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530590 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530591 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530592 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
593 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Suchitra Vemuriebf65322020-04-03 10:09:41 -0700594 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
595 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
596 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
597 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
598 ... ${dst['container_name']}
599 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
600 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
601 END
602
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700603Verify OLT Soft Reboot
604 [Documentation] Test soft reboot of the OLT using voltctl command
605 [Tags] VOL-2745 OLTSoftReboot functional
606 [Setup] Start Logging OLTSoftReboot
607 [Teardown] Run Keywords Collect Logs
608 ... AND Stop Logging OLTSoftReboot
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800609 FOR ${I} IN RANGE 0 ${olt_count}
610 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
611 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
612 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 360s 5s
613 ... Validate OLT Device ENABLED ACTIVE
614 ... REACHABLE ${olt_serial_number}
615 # Reboot the OLT using "voltctl device reboot" command
616 Reboot Device ${olt_device_id}
Andrea Campanella9d8d3a52021-06-17 12:18:23 +0200617 # Wait for the OLT to actually go down
618 Wait Until Keyword Succeeds 360s 5s Validate OLT Device ENABLED UNKNOWN UNREACHABLE
619 ... ${olt_serial_number}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800620 END
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700621 #Verify that ping fails
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800622 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700623 ${src}= Set Variable ${hosts.src[${I}]}
624 ${dst}= Set Variable ${hosts.dst[${I}]}
625 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +0530626 ... Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700627 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
628 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
629 END
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700630 # Check OLT states
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800631 FOR ${I} IN RANGE 0 ${olt_count}
632 ${olt_serial_number}= Get From Dictionary ${list_olts}[${I}] sn
633 ${olt_ssh_ip}= Get From Dictionary ${list_olts}[${I}] sship
634 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
635 # Wait for the OLT to come back up
636 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds 120s 10s
637 ... Check Remote System Reachability True ${olt_ssh_ip}
638 # Check OLT states
Hardik Windlass58b23a32022-05-10 11:06:48 +0530639 Wait Until Keyword Succeeds 360s 5s
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800640 ... Validate OLT Device ENABLED ACTIVE
641 ... REACHABLE ${olt_serial_number}
642 END
643 # Waiting extra time for the ONUs to come up
644 Sleep 60s
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700645 #Check after reboot that ONUs are active, authenticated/DHCP/pingable
646 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530647 Perform Sanity Test
Suchitra Vemuri19e942f2020-08-11 18:04:28 -0700648
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700649Verify restart ofagent container before subscriber is provisioned
650 [Documentation] Restart ofagent container before subscriber is provisioned.
651 [Tags] functional VOL-2962 ofagentRestart2
652 [Setup] Start Logging ofagentRestart2
653 [Teardown] Run Keywords Collect Logs
Suchitra Vemuri8130be82020-09-15 16:47:58 -0700654 ... AND Stop Logging ofagentRestart2
Matteo Scandolo01fefb22021-12-14 14:33:56 -0800655 ... AND Scale K8s Deployment ${NAMESPACE} ${STACK_NAME}-voltha-ofagent 1
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800656 Delete All Devices And Verify
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700657 setup
658 Run Keyword If ${has_dataplane} Clean Up Linux
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800659 FOR ${I} IN RANGE 0 ${olt_count}
660 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
661 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
662 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
663 ... ${olt_serial_number}
664 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
665 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
666 END
667 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700668 ${src}= Set Variable ${hosts.src[${I}]}
669 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800670 ${of_id}= Get ofID From OLT List ${src['olt']}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700671 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000672 ... ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530673 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700674 # Bring up the device and verify it authenticates
675 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
Hardik Windlass8b446492021-06-10 06:25:23 +0000676 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed by_dev_id=True
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700677 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700678 ... ${ONOS_SSH_PORT} ${of_id} ${onu_port}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700679 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
680 ... /tmp/wpa ${src['dp_iface_name']} log
681 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
682 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
683 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
684 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
685 ... ${ONOS_SSH_PORT} ${onu_port}
686 END
687 # Restart POD ofagent
688 ${waitforRestart} Set Variable 120s
689 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
690 Log ${podStatusOutput}
691 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
692 ${podName} Set Variable ofagent
TorstenThieme37165402021-09-03 11:39:40 +0000693 Restart Pod By Label ${NAMESPACE} app ${podName}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700694 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ofagent ${NAMESPACE}
695 ... Running
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800696 FOR ${I} IN RANGE 0 ${num_all_onus}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700697 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000698 #Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700699 #... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
700 ${src}= Set Variable ${hosts.src[${I}]}
701 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800702 ${of_id}= Get ofID From OLT List ${src['olt']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530703 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Hardik Windlass1ed2eee2021-06-25 09:51:03 +0000704 ... Get ONU Port in ONOS ${src['onu']} ${of_id} ${src['uni_id']}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530705 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700706 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
TorstenThieme4e2168e2021-06-22 14:01:47 +0000707 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700708 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
709 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
710 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
711 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
712 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
713 ... ${dst['container_name']}
714 # Verify DHCP-Allocations
Hardik Windlass58b23a32022-05-10 11:06:48 +0530715 Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700716 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri10e737e2020-08-14 14:59:10 -0700717 END
718
Hardik Windlassc082c422021-03-16 15:44:10 +0000719Verify OLT Grpc Disconnection
720 [Documentation] Restarts OLT Grpc Server and verifies everything works as before without any system disruption.
721 [Tags] functional VOL-3904 restartGrpcServer bbsim
722 [Setup] Start Logging restartGrpcServer
723 [Teardown] Run Keywords Collect Logs
724 ... AND Stop Logging restartGrpcServer
725 Delete All Devices And Verify
726 Setup
727 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530728 Perform Sanity Test
Hardik Windlassc082c422021-03-16 15:44:10 +0000729 FOR ${J} IN RANGE 0 ${num_olts}
730 ${bbsim_rel}= Catenate SEPARATOR= bbsim ${J}
731 ${bbsim_pod}= Get Pod Name By Label ${NAMESPACE} release ${bbsim_rel}
732 Restart Grpc Server ${NAMESPACE} ${bbsim_pod} 5
733 END
734 # Repeat sanity test without subscriber changes
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700735 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test ${suppressaddsubscriber}
Elia Battiston00e586b2022-02-15 10:15:50 +0100736 # Without this sleep there are cases in which the request to disable the device
737 # comes before the Grpc server is restarted and is never executed.
738 # Checking BBSIM's connection status before proceeding would be better, but it is not currently exposed
739 Sleep 15s
Hardik Windlassc082c422021-03-16 15:44:10 +0000740 # Additional Verification
741 Wait Until Keyword Succeeds ${timeout} 2s Delete All Devices and Verify
742 Setup
743 Run Keyword If ${has_dataplane} Clean Up Linux
Hardik Windlass58b23a32022-05-10 11:06:48 +0530744 Perform Sanity Test
Hardik Windlassc082c422021-03-16 15:44:10 +0000745
Hema123aff52020-05-26 15:41:47 +0530746Verify ONU Soft Reboot
747 [Documentation] Test soft reboot of the ONU using voltctl command
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800748 [Tags] VOL-1957 ONUSoftReboot functional notready
Hema123aff52020-05-26 15:41:47 +0530749 [Setup] Start Logging ONUSoftReboot
750 #... AND Setup
Andy Bavier4a8450e2020-02-04 08:58:37 -0700751 [Teardown] Run Keywords Collect Logs
Hema123aff52020-05-26 15:41:47 +0530752 ... AND Stop Logging ONUSoftReboot
753 #... AND Delete Device and Verify
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800754 FOR ${I} IN RANGE 0 ${num_all_onus}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000755 ${src}= Set Variable ${hosts.src[${I}]}
756 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800757 ${of_id}= Get ofID From OLT List ${src['olt']}
TorstenThiemedda37652021-11-11 16:22:05 +0000758 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530759 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800760 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
Hardik Windlass78bb0fd2022-06-21 15:11:54 +0530761 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Hema123aff52020-05-26 15:41:47 +0530762 Reboot Device ${onu_device_id}
763 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
Hardik Windlass58b23a32022-05-10 11:06:48 +0530764 ... Wait Until Keyword Succeeds ${timeout} 2s
Hema123aff52020-05-26 15:41:47 +0530765 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
766 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
767 # Remove Subscriber Access (To replicate ATT workflow)
TorstenThieme4e2168e2021-06-22 14:01:47 +0000768 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
Hema123aff52020-05-26 15:41:47 +0530769 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Hardik Windlassb75cf1b2021-01-18 06:59:09 +0000770 Verify ping is successful except for given device ${num_all_onus} ${src['onu']}
Hema123aff52020-05-26 15:41:47 +0530771 Sleep 40s
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800772 # Check ONU port is Enabled in ONOS
773 Run Keyword And Continue On Failure Wait Until Keyword Succeeds 120s 2s
Hardik Windlass39015672021-07-05 05:48:08 +0000774 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800775 # Verify EAPOL flows are added for the ONU port
776 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
Matteo Scandolo7e519fb2021-08-13 11:35:16 -0700777 ... Verify Eapol Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id} ${onu_port}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800778 # Verify ONU state in voltha
Hardik Windlass58b23a32022-05-10 11:06:48 +0530779 Wait Until Keyword Succeeds 360s 5s Validate Device
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800780 ... ENABLED ACTIVE REACHABLE
781 ... ${src['onu']} onu=True onu_reason=omci-flows-pushed
Hema123aff52020-05-26 15:41:47 +0530782 Run Keyword If ${has_dataplane} Clean Up Linux
783 # Perform Authentication
784 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
785 ... /tmp/wpa ${src['dp_iface_name']} log
786 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
787 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
788 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530789 Wait Until Keyword Succeeds ${timeout} 2
Hema123aff52020-05-26 15:41:47 +0530790 ... Verify ONU in AAA-Users ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530791 Wait Until Keyword Succeeds ${timeout} 2
TorstenThieme4e2168e2021-06-22 14:01:47 +0000792 ... Execute ONOS CLI Command use single connection ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
Hema123aff52020-05-26 15:41:47 +0530793 ... volt-add-subscriber-access ${of_id} ${onu_port}
794 # Verify that no pending flows exist for the ONU port
795 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
796 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Hardik Windlass21807632020-04-14 16:24:55 +0530797 # Verify subscriber access flows are added for the ONU port
Hardik Windlass58b23a32022-05-10 11:06:48 +0530798 Wait Until Keyword Succeeds ${timeout} 5s
Hardik Windlass21807632020-04-14 16:24:55 +0530799 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
800 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
Hema123aff52020-05-26 15:41:47 +0530801 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
802 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuricd2f64f2020-02-18 18:30:27 -0800803 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Hema123aff52020-05-26 15:41:47 +0530804 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
805 ... ${dst['container_name']}
Hardik Windlass58b23a32022-05-10 11:06:48 +0530806 Wait Until Keyword Succeeds ${timeout} 2s
Hema123aff52020-05-26 15:41:47 +0530807 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
Gayathri.Selvanf68ea4b2020-02-03 07:36:39 +0000808 END
Suchitra Vemuria6879aa2020-11-03 11:03:11 -0800809
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800810
811*** Keywords ***
812Setup Suite
813 [Documentation] Set up the test suite
814 Common Test Suite Setup
HungWei Chiu1408fb92020-03-03 19:43:30 -0500815 #power_switch.robot needs it to support different vendor's power switch
Hung-Wei Chiufcc6d8e2020-03-10 12:05:11 -0700816 ${switch_type}= Get Variable Value ${web_power_switch.type}
817 Run Keyword If "${switch_type}"!="" Set Global Variable ${powerswitch_type} ${switch_type}
Suchitra Vemurif7a033c2020-02-26 17:22:41 -0800818
819Clear All Devices Then Create New Device
820 [Documentation] Remove any devices from VOLTHA and ONOS
821 # Remove all devices from voltha and nos
822 Delete All Devices and Verify
823 # Execute normal test Setup Keyword
824 Setup
TorstenThieme4e2168e2021-06-22 14:01:47 +0000825
826Teardown Suite
827 [Documentation] Clean up ONOS SSH connections
828 Close All ONOS SSH Connections
Hardik Windlass81375222021-12-02 10:48:01 +0000829 Run Keyword If ${has_dataplane} Clean Up All Nodes