blob: 70d16a0150ad16f0e23a01493d9c5fecbc15e1db [file] [log] [blame]
David Bainbridgef81cd642019-11-20 00:14:47 +00001# Copyright 2017 - present Open Networking Foundation
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -07002#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Matteo Scandolo1294aeb2019-09-24 16:20:32 -070014# FIXME Can we use the same test against BBSim and Hardware?
15
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070016*** Settings ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070017Documentation Test various end-to-end scenarios
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070018Suite Setup Setup Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070019Test Setup Setup
20Test Teardown Teardown
Andy Bavierba9866b2019-10-11 07:11:53 -070021Suite Teardown Teardown Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070022Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
Andy Bavierba9866b2019-10-11 07:11:53 -070027Library ../../libraries/DependencyLibrary.py
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070028Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
David Bainbridgef81cd642019-11-20 00:14:47 +000030Resource ../../libraries/voltha.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070031Resource ../../libraries/utils.robot
Andy Bavierba9866b2019-10-11 07:11:53 -070032Resource ../../libraries/k8s.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070033Resource ../../variables/variables.robot
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070034
35*** Variables ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070036${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
Zack Williamsa8fe75a2020-01-10 14:25:27 -070043${NAMESPACE} voltha
suraj gourd64356b2019-11-07 13:26:20 +000044# For below variable value, using deployment name as using grep for
45# parsing radius pod name, we can also use full radius pod name
Zack Williamsa8fe75a2020-01-10 14:25:27 -070046${RESTART_POD_NAME} radius
Andy Bavierba9866b2019-10-11 07:11:53 -070047${timeout} 60s
Zack Williamsec53a1b2019-09-16 15:50:52 -070048${of_id} 0
49${logical_id} 0
Andy Bavierba9866b2019-10-11 07:11:53 -070050${has_dataplane} True
51${external_libs} True
52${teardown_device} False
Zack Williamsa8fe75a2020-01-10 14:25:27 -070053${scripts} ../../scripts
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070054
55*** Test Cases ***
56Sanity E2E Test for OLT/ONU on POD
57 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
58 ... Validate successful authentication/DHCP/E2E ping for the tech profile that is used
Andy Bavierba9866b2019-10-11 07:11:53 -070059 [Tags] sanity test1
Suchitra Vemuri6db89412019-11-14 14:52:54 -080060 [Teardown] NONE
Suchitra Vemuric5295a32019-12-15 20:32:04 -080061 Run Keyword If ${has_dataplane} Clean Up Linux
Zack Williamsa8fe75a2020-01-10 14:25:27 -070062 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuri582574b2019-12-10 21:11:54 -080063 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070064
Suchitra Vemuri6db89412019-11-14 14:52:54 -080065Test Disable and Enable ONU
66 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
67 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
68 ... Perform disable on the ONUs and validate that the pings do not succeed
69 ... Perform enable on the ONUs and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -080070 [Tags] functional DisableEnableONU released
Suchitra Vemuri6db89412019-11-14 14:52:54 -080071 [Setup] None
suraj gour7f6d5fe2019-11-29 10:56:35 +000072 [Teardown] None
Suchitra Vemuri6db89412019-11-14 14:52:54 -080073 FOR ${I} IN RANGE 0 ${num_onus}
74 ${src}= Set Variable ${hosts.src[${I}]}
75 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri6db89412019-11-14 14:52:54 -080076 ${onu_device_id}= Get Device ID From SN ${src['onu']}
77 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
78 ... ${of_id}
79 Disable Device ${onu_device_id}
Suchitra Vemuri15e15252020-01-20 17:38:49 -080080 #Test Devices Disabled in VOLTHA Id=${onu_device_id}
Andy Bavier33e6dd32020-01-16 13:35:20 -070081 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
82 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070083 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
84 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri6db89412019-11-14 14:52:54 -080085 Enable Device ${onu_device_id}
86 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
87 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -070088 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
89 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070090 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
91 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
92 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
93 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri6db89412019-11-14 14:52:54 -080094 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -070095 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri6db89412019-11-14 14:52:54 -080096
Suchitra Vemuri5994cd12019-12-17 22:20:55 -080097Test Subscriber Delete and Add
98 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
99 ... Assuming that all the ONUs are authenticated/DHCP/pingable
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700100 ... Delete a subscriber and validate that the pings do not succeed
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800101 ... Re-add the subscriber and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800102 [Tags] functional SubAddDelete released
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800103 [Setup] None
104 [Teardown] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800105 FOR ${I} IN RANGE 0 ${num_onus}
106 ${src}= Set Variable ${hosts.src[${I}]}
107 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800108 ${onu_device_id}= Get Device ID From SN ${src['onu']}
109 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
110 ... ${of_id}
111 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
112 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
113 Sleep 10s
Andy Bavier33e6dd32020-01-16 13:35:20 -0700114 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
115 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700116 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
117 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800118 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
119 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
120 Sleep 10s
121 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
122 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700123 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
124 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700125 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
126 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
127 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
128 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800129 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700130 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800131
suraj gourd64356b2019-11-07 13:26:20 +0000132Check OLT/ONU Authentication After Radius Pod Restart
suraj gour7f6d5fe2019-11-29 10:56:35 +0000133 [Documentation] After radius restart, triggers reassociation, checks status and
134 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
135 ... wpa supplicant is running in background hence it is recommended to remove
136 ... teardown from previous test or uncomment 'Teardown None'.
137 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800138 [Tags] functional RadiusRestart released
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700139 [Setup] None
140 [Teardown] None
suraj gourd64356b2019-11-07 13:26:20 +0000141 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
suraj gourd64356b2019-11-07 13:26:20 +0000142 FOR ${I} IN RANGE 0 ${num_onus}
143 ${src}= Set Variable ${hosts.src[${I}]}
144 ${dst}= Set Variable ${hosts.dst[${I}]}
suraj gourd64356b2019-11-07 13:26:20 +0000145 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700146 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
147 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
148 Wait Until Keyword Succeeds ${timeout} 2s
149 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
150 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
151 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
152 ... ${src['ip']} ${src['user']} ${src['pass']}
suraj gourd64356b2019-11-07 13:26:20 +0000153 ... ${src['container_type']} ${src['container_name']}
154 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700155 ... ${ONOS_SSH_PORT} ${onu_port}
156 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
157 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
158 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
159 ... ${src['ip']} ${src['user']} ${src['pass']}
160 ... ${src['container_type']} ${src['container_name']}
161 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
162 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
suraj gourd64356b2019-11-07 13:26:20 +0000163 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
164 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700165 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
166 Run Keyword and Ignore Error Collect Logs
suraj gourd64356b2019-11-07 13:26:20 +0000167 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700168 Run Keyword and Ignore Error Collect Logs
suraj gourd64356b2019-11-07 13:26:20 +0000169
suraj gourd5cfdbb2019-12-13 12:44:55 +0000170Check DHCP attempt fails when subscriber is not added
171 [Documentation] Validates when removed subscriber access, DHCP attempt, ping fails and
172 ... when again added subscriber access, DHCP attempt, ping succeeds
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700173 ... Assuming that test1 or sanity test was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800174 [Tags] functional SubsRemoveDHCP released
suraj gourd5cfdbb2019-12-13 12:44:55 +0000175 [Setup] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800176 [Teardown] None
suraj gourd5cfdbb2019-12-13 12:44:55 +0000177 FOR ${I} IN RANGE 0 ${num_onus}
178 ${src}= Set Variable ${hosts.src[${I}]}
179 ${dst}= Set Variable ${hosts.dst[${I}]}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700180 ${onu_device_id}= Get Device ID From SN ${src['onu']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000181 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
182 ... ${of_id}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800183 Run Keyword And Ignore Error Login And Run Command On Remote System killall dhclient ${src['ip']}
184 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
185 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
186 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000187 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
188 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700189 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800190 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
191 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700192 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds ${timeout} 2s
193 ... Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
194 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
195 Run Keyword And Ignore Error Login And Run Command On Remote System
196 ... ifconfig | grep -A 10 ens ${src['ip']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800197 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000198 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping False
199 ... False ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
200 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
201 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
202 ... ${dst['container_name']}
203 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
204 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
205 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
206 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
207 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
208 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
209 ... ${dst['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800210 Run Keyword and Ignore Error Collect Logs
211 END
212 Run Keyword and Ignore Error Collect Logs
213
suraj gour1ecfae92019-12-20 15:11:40 +0000214Check ONU adapter crash not forcing authentication again
215 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700216 ... message count to make sure auth not started again and validates EAP status and ping.
suraj gour1ecfae92019-12-20 15:11:40 +0000217 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800218 [Tags] functional ONUAdaptCrash notready
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700219 [Setup] None
220 [Teardown] None
suraj gour1ecfae92019-12-20 15:11:40 +0000221 @{before_list}= Create List
222 @{after_list}= Create List
223 FOR ${I} IN RANGE 0 ${num_onus}
224 ${src}= Set Variable ${hosts.src[${I}]}
225 ${dst}= Set Variable ${hosts.dst[${I}]}
226 ${onu_device_id}= Get Device ID From SN ${src['onu']}
227 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
228 ... ${of_id}
229 ${before}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
230 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
231 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700232 Append To List ${before_list} ${before}
suraj gour1ecfae92019-12-20 15:11:40 +0000233 END
234 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} adapter-open-onu
235 Wait Until Keyword Succeeds ${timeout} 2s Validate Pod Status ${podName} ${NAMESPACE}
236 ... Running
237 FOR ${I} IN RANGE 0 ${num_onus}
238 ${src}= Set Variable ${hosts.src[${I}]}
239 ${dst}= Set Variable ${hosts.dst[${I}]}
240 ${onu_device_id}= Get Device ID From SN ${src['onu']}
241 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
242 ... ${of_id}
243 ${after}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
244 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
245 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700246 Append To List ${after_list} ${after}
247 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
248 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
249 ... ${src['container_type']} ${src['container_name']}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700250 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700251 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
252 ... Wait Until Keyword Succeeds 60s 2s Check Ping
suraj gour1ecfae92019-12-20 15:11:40 +0000253 ... True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} ${src['user']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700254 ... ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gour1ecfae92019-12-20 15:11:40 +0000255 END
256 Lists Should Be Equal ${after_list} ${before_list}
257 Log ${after_list}
258 Log ${before_list}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700259 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800260
261Test Disable and Enable ONU scenario for ATT workflow
262 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
263 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
264 ... Perform disable on the ONUs, call volt-remove-subscriber and validate that the pings do not succeed
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700265 ... Perform enable on the ONUs, authentication check, volt-add-subscriber-access and
266 ... validate that the pings are successful
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800267 ... VOL-2284
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800268 [Tags] functional ATT_DisableEnableONU
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800269 [Setup] None
270 #[Teardown] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800271 FOR ${I} IN RANGE 0 ${num_onus}
272 ${src}= Set Variable ${hosts.src[${I}]}
273 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800274 ${onu_device_id}= Get Device ID From SN ${src['onu']}
275 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
276 ... ${of_id}
277 Disable Device ${onu_device_id}
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800278 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800279 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
280 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700281 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
282 ... Wait Until Keyword Succeeds 60s 2s Check Ping
283 ... False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
284 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800285 ... ELSE sleep 60s
286 Enable Device ${onu_device_id}
287 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
288 ... ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700289 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
290 ... Validate Authentication After Reassociate True
291 ... ${src['dp_iface_name']} ${src['ip']} ${src['user']} ${src['pass']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800292 ... ${src['container_type']} ${src['container_name']}
293 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700294 ... ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800295 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
296 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
297 Sleep 10s
298 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
Andy Bavier08ae5852019-12-19 09:12:42 -0700299 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800300 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
301 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
302 ... ${dst['container_name']}
303 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
304 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800305 Run Keyword and Ignore Error Collect Logs
suraj gourd5cfdbb2019-12-13 12:44:55 +0000306 END
307 Run Keyword and Ignore Error Collect Logs
308
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800309Delete OLT, ReAdd OLT and Perform Sanity Test
310 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
311 ... Disable and Delete the OLT
312 ... Create/Enable the same OLT again
313 ... Validate authentication/DHCP/E2E pings succeed for all the ONUs connected to the OLT
Suchitra Vemuri2fa9bba2020-01-22 17:38:48 -0800314 [Tags] functional DeleteOLT
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800315 [Setup] None
316 [Teardown] NONE
317 Run Keyword If ${has_dataplane} Clean Up Linux
318 Run Keyword If ${has_dataplane} Delete Device and Verify
319 Run Keyword and Ignore Error Collect Logs
320 # Recreate the OLT
321 Run Keyword If ${has_dataplane} Setup
322 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
323 Run Keyword and Ignore Error Collect Logs
324
Hema783279b2020-01-22 15:37:37 +0530325Test disable ONUs and OLT then delete ONUs and OLT
326 [Documentation] On deployed POD, disable the ONU, disable the OLT and then delete ONU and OLT.
327 ... This TC is to confirm that ONU removal is not impacting OLT
328 ... Devices will be removed during the execution of this TC
329 ... so calling setup at the end to add the devices back to avoid the confusion.
330 [Tags] VOL-2354 DisableDeleteONUandOLT notready
331 [Setup] NONE
332 [Teardown] None
333 FOR ${I} IN RANGE 0 ${num_onus}
334 ${src}= Set Variable ${hosts.src[${I}]}
335 ${dst}= Set Variable ${hosts.dst[${I}]}
336 ${onu_device_id}= Get Device ID From SN ${src['onu']}
337 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE
338 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
339 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
340 ... REACHABLE ${olt_serial_number}
341 ${rc} ${output}= Run and Return Rc and Output
342 ... ${VOLTCTL_CONFIG}; voltctl device disable ${onu_device_id}
343 Should Be Equal As Integers ${rc} 0
344 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED UNKNOWN
345 ... REACHABLE ${src['onu']} onu=false
346 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
347 ... REACHABLE ${olt_serial_number}
348 END
349 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
350 Should Be Equal As Integers ${rc} 0
351 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
352 ... ${olt_serial_number}
353 FOR ${I} IN RANGE 0 ${num_onus}
354 ${src}= Set Variable ${hosts.src[${I}]}
355 ${dst}= Set Variable ${hosts.dst[${I}]}
356 ${onu_device_id}= Get Device ID From SN ${src['onu']}
357 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED DISCOVERED
358 ... UNREACHABLE ${src['onu']} onu=false
359 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${onu_device_id}
360 Should Be Equal As Integers ${rc} 0
361 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN
362 ... REACHABLE ${olt_serial_number}
363 END
364 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${olt_device_id}
365 Should Be Equal As Integers ${rc} 0
366 Wait Until Keyword Succeeds ${timeout} 5s Test Empty Device List
367 #Adding setup here to add the devices back since this TC removes the devices
368 Run Keyword If ${has_dataplane} sleep 180s
369 setup
370
David Bainbridgef81cd642019-11-20 00:14:47 +0000371Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
372 [Documentation] Deploys an device instance and waits for it to authenticate. After
373 ... authentication is successful the rw-core deployment is scaled to 0 instances to
374 ... simulate a POD crash. The test then scales the rw-core back to a single instance
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700375 ... and configures ONOS for access. The test succeeds if the device is able to
David Bainbridgef81cd642019-11-20 00:14:47 +0000376 ... complete the DHCP sequence.
377 [Tags] bbsim rwcore-restart
378 [Setup] Clear All Devices Then Create New Device
379 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
380 Set Global Variable ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000381 FOR ${I} IN RANGE 0 ${num_onus}
382 ${src}= Set Variable ${hosts.src[${I}]}
383 ${dst}= Set Variable ${hosts.dst[${I}]}
384 ${onu_device_id}= Get Device ID From SN ${src['onu']}
385 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
386 ... ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000387 # Bring up the device and verify it authenticates
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700388 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
David Bainbridgef81cd642019-11-20 00:14:47 +0000389 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
390 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
391 ... ${ONOS_SSH_PORT} ${onu_port}
392 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
393 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
394 ... ${src['container_type']} ${src['container_name']}
395 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700396 ... ${ONOS_SSH_PORT} ${onu_port}
David Bainbridgef81cd642019-11-20 00:14:47 +0000397 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
398 Scale K8s Deployment voltha voltha-rw-core 0
399 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
400 # Ensure the ofagent POD goes "not-ready" as expected
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700401 Wait Until keyword Succeeds ${timeout} 2s
402 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
David Bainbridgef81cd642019-11-20 00:14:47 +0000403 # Scale up the core deployment and make sure both it and the ofagent deployment are back
404 Scale K8s Deployment voltha voltha-rw-core 1
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700405 Wait Until Keyword Succeeds ${timeout} 2s
406 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
407 Wait Until Keyword Succeeds ${timeout} 2s
408 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
David Bainbridgef81cd642019-11-20 00:14:47 +0000409 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
410 # so restart the port forwarding for the API service
411 Restart VOLTHA Port Foward voltha-api-minimal
David Bainbridgef81cd642019-11-20 00:14:47 +0000412 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
413 # represents system connectivity being restored
414 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
415 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000416 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
417 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
418 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
419 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
420 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
421 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
422 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
423 ... ${dst['container_name']}
424 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
425 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
426 END
427
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700428*** Keywords ***
429Setup Suite
Andy Bavier88cd9f62019-11-26 16:22:33 -0700430 [Documentation] Set up the test suite
431 Common Test Suite Setup
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700432
David Bainbridgef81cd642019-11-20 00:14:47 +0000433Clear All Devices Then Create New Device
434 [Documentation] Remove any devices from VOLTHA and ONOS
David Bainbridgef81cd642019-11-20 00:14:47 +0000435 # Remove all devices from voltha and nos
436 Delete All Devices and Verify
David Bainbridgef81cd642019-11-20 00:14:47 +0000437 # Execute normal test Setup Keyword
438 Setup
439
Suchitra Vemuri582574b2019-12-10 21:11:54 -0800440Collect Logs
441 [Documentation] Collect Logs from voltha and onos cli for various commands
Suchitra Vemuri1a970a62019-11-26 12:52:16 -0800442 Run Keyword and Ignore Error Get Device List from Voltha
443 Run Keyword and Ignore Error Get Device Output from Voltha ${olt_device_id}
444 Run Keyword and Ignore Error Get Logical Device Output from Voltha ${logical_id}
Andy Bavierba9866b2019-10-11 07:11:53 -0700445 Run Keyword If ${external_libs} Get ONOS Status ${k8s_node_ip}
Andy Bavierba9866b2019-10-11 07:11:53 -0700446
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700447Clean Up Linux
448 [Documentation] Kill processes and clean up interfaces on src+dst servers
Suchitra Vemuri8a9c3782019-10-23 12:43:01 -0700449 FOR ${I} IN RANGE 0 ${num_onus}
450 ${src}= Set Variable ${hosts.src[${I}]}
451 ${dst}= Set Variable ${hosts.dst[${I}]}
452 Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
453 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
454 Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src['ip']}
455 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
456 Run Keyword If '${dst['ip']}' != '${None}' Run Keyword And Ignore Error
457 ... Kill Linux Process [d]hcpd ${dst['ip']} ${dst['user']}
458 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
459 Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
460 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
461 Run Keyword If '${dst['ip']}' != '${None}' Delete Interface on Remote Host
462 ... ${dst['dp_iface_name']}.${src['s_tag']} ${dst['ip']} ${dst['user']} ${dst['pass']}
463 ... ${dst['container_type']} ${dst['container_name']}
464 END
Andy Bavierba9866b2019-10-11 07:11:53 -0700465