blob: c24b6833eb9ecb13355a6924f4c44293a0284aba [file] [log] [blame]
David Bainbridgef81cd642019-11-20 00:14:47 +00001# Copyright 2017 - present Open Networking Foundation
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -07002#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
Matteo Scandolo1294aeb2019-09-24 16:20:32 -070014# FIXME Can we use the same test against BBSim and Hardware?
15
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070016*** Settings ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070017Documentation Test various end-to-end scenarios
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070018Suite Setup Setup Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070019Test Setup Setup
20Test Teardown Teardown
Andy Bavierba9866b2019-10-11 07:11:53 -070021Suite Teardown Teardown Suite
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070022Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
Andy Bavierba9866b2019-10-11 07:11:53 -070027Library ../../libraries/DependencyLibrary.py
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070028Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
David Bainbridgef81cd642019-11-20 00:14:47 +000030Resource ../../libraries/voltha.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070031Resource ../../libraries/utils.robot
Andy Bavierba9866b2019-10-11 07:11:53 -070032Resource ../../libraries/k8s.robot
Suchitra Vemurib8ed2d52019-09-30 13:22:51 -070033Resource ../../variables/variables.robot
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070034
35*** Variables ***
Zack Williamsec53a1b2019-09-16 15:50:52 -070036${POD_NAME} flex-ocp-cord
37${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
38${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
39#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
40${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
41${HELM_CHARTS_DIR} ~/helm-charts
42${VOLTHA_POD_NUM} 8
Zack Williamsa8fe75a2020-01-10 14:25:27 -070043${NAMESPACE} voltha
suraj gourd64356b2019-11-07 13:26:20 +000044# For below variable value, using deployment name as using grep for
45# parsing radius pod name, we can also use full radius pod name
Zack Williamsa8fe75a2020-01-10 14:25:27 -070046${RESTART_POD_NAME} radius
Andy Bavierba9866b2019-10-11 07:11:53 -070047${timeout} 60s
Zack Williamsec53a1b2019-09-16 15:50:52 -070048${of_id} 0
49${logical_id} 0
Andy Bavierba9866b2019-10-11 07:11:53 -070050${has_dataplane} True
51${external_libs} True
52${teardown_device} False
Zack Williamsa8fe75a2020-01-10 14:25:27 -070053${scripts} ../../scripts
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070054
55*** Test Cases ***
56Sanity E2E Test for OLT/ONU on POD
57 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
58 ... Validate successful authentication/DHCP/E2E ping for the tech profile that is used
Andy Bavierba9866b2019-10-11 07:11:53 -070059 [Tags] sanity test1
Suchitra Vemuri6db89412019-11-14 14:52:54 -080060 [Teardown] NONE
Suchitra Vemuric5295a32019-12-15 20:32:04 -080061 Run Keyword If ${has_dataplane} Clean Up Linux
Zack Williamsa8fe75a2020-01-10 14:25:27 -070062 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
Suchitra Vemuri582574b2019-12-10 21:11:54 -080063 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -070064
Suchitra Vemuri6db89412019-11-14 14:52:54 -080065Test Disable and Enable ONU
66 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
67 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
68 ... Perform disable on the ONUs and validate that the pings do not succeed
69 ... Perform enable on the ONUs and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -080070 [Tags] functional DisableEnableONU released
Suchitra Vemuri6db89412019-11-14 14:52:54 -080071 [Setup] None
suraj gour7f6d5fe2019-11-29 10:56:35 +000072 [Teardown] None
Suchitra Vemuri6db89412019-11-14 14:52:54 -080073 FOR ${I} IN RANGE 0 ${num_onus}
74 ${src}= Set Variable ${hosts.src[${I}]}
75 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri6db89412019-11-14 14:52:54 -080076 ${onu_device_id}= Get Device ID From SN ${src['onu']}
77 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
78 ... ${of_id}
79 Disable Device ${onu_device_id}
Suchitra Vemuri15e15252020-01-20 17:38:49 -080080 #Test Devices Disabled in VOLTHA Id=${onu_device_id}
Andy Bavier33e6dd32020-01-16 13:35:20 -070081 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
82 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070083 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
84 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri6db89412019-11-14 14:52:54 -080085 Enable Device ${onu_device_id}
86 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
87 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -070088 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
89 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -070090 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
91 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
92 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
93 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri6db89412019-11-14 14:52:54 -080094 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -070095 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri6db89412019-11-14 14:52:54 -080096
Suchitra Vemuri5994cd12019-12-17 22:20:55 -080097Test Subscriber Delete and Add
98 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
99 ... Assuming that all the ONUs are authenticated/DHCP/pingable
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700100 ... Delete a subscriber and validate that the pings do not succeed
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800101 ... Re-add the subscriber and validate that the pings are successful
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800102 [Tags] functional SubAddDelete released
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800103 [Setup] None
104 [Teardown] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800105 FOR ${I} IN RANGE 0 ${num_onus}
106 ${src}= Set Variable ${hosts.src[${I}]}
107 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800108 ${onu_device_id}= Get Device ID From SN ${src['onu']}
109 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
110 ... ${of_id}
111 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
112 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
113 Sleep 10s
Andy Bavier33e6dd32020-01-16 13:35:20 -0700114 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
115 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700116 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
117 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800118 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
119 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
120 Sleep 10s
121 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
122 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700123 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
124 ... Wait Until Keyword Succeeds 60s 2s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700125 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
126 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
127 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
128 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800129 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700130 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800131
suraj gourd64356b2019-11-07 13:26:20 +0000132Check OLT/ONU Authentication After Radius Pod Restart
suraj gour7f6d5fe2019-11-29 10:56:35 +0000133 [Documentation] After radius restart, triggers reassociation, checks status and
134 ... authentication, validates dhcp and ping. Note : wpa reassociate works only when
135 ... wpa supplicant is running in background hence it is recommended to remove
136 ... teardown from previous test or uncomment 'Teardown None'.
137 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800138 [Tags] functional RadiusRestart released
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700139 [Setup] None
140 [Teardown] None
suraj gourd64356b2019-11-07 13:26:20 +0000141 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} ${RESTART_POD_NAME}
suraj gourd64356b2019-11-07 13:26:20 +0000142 FOR ${I} IN RANGE 0 ${num_onus}
143 ${src}= Set Variable ${hosts.src[${I}]}
144 ${dst}= Set Variable ${hosts.dst[${I}]}
suraj gourd64356b2019-11-07 13:26:20 +0000145 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700146 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
147 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
148 Wait Until Keyword Succeeds ${timeout} 2s
149 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
150 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
151 ... Validate Authentication After Reassociate True ${src['dp_iface_name']}
152 ... ${src['ip']} ${src['user']} ${src['pass']}
suraj gourd64356b2019-11-07 13:26:20 +0000153 ... ${src['container_type']} ${src['container_name']}
154 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700155 ... ${ONOS_SSH_PORT} ${onu_port}
156 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
157 ... Validate DHCP and Ping True True ${src['dp_iface_name']}
158 ... ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
159 ... ${src['ip']} ${src['user']} ${src['pass']}
160 ... ${src['container_type']} ${src['container_name']}
161 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']}
162 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
suraj gourd64356b2019-11-07 13:26:20 +0000163 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
164 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700165 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
166 Run Keyword and Ignore Error Collect Logs
suraj gourd64356b2019-11-07 13:26:20 +0000167 END
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700168 Run Keyword and Ignore Error Collect Logs
suraj gourd64356b2019-11-07 13:26:20 +0000169
suraj gourd5cfdbb2019-12-13 12:44:55 +0000170Check DHCP attempt fails when subscriber is not added
171 [Documentation] Validates when removed subscriber access, DHCP attempt, ping fails and
172 ... when again added subscriber access, DHCP attempt, ping succeeds
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700173 ... Assuming that test1 or sanity test was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800174 [Tags] functional SubsRemoveDHCP released
suraj gourd5cfdbb2019-12-13 12:44:55 +0000175 [Setup] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800176 [Teardown] None
suraj gourd5cfdbb2019-12-13 12:44:55 +0000177 FOR ${I} IN RANGE 0 ${num_onus}
178 ${src}= Set Variable ${hosts.src[${I}]}
179 ${dst}= Set Variable ${hosts.dst[${I}]}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700180 ${onu_device_id}= Get Device ID From SN ${src['onu']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000181 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
182 ... ${of_id}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800183 Run Keyword And Ignore Error Login And Run Command On Remote System killall dhclient ${src['ip']}
184 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
185 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
186 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000187 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
188 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700189 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800190 Run Keyword And Ignore Error Login And Run Command On Remote System ps -ef | grep dhclient ${src['ip']}
191 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700192 Run Keyword If ${has_dataplane} Wait Until Keyword Succeeds ${timeout} 2s
193 ... Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
194 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
195 Run Keyword And Ignore Error Login And Run Command On Remote System
196 ... ifconfig | grep -A 10 ens ${src['ip']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800197 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gourd5cfdbb2019-12-13 12:44:55 +0000198 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping False
199 ... False ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
200 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
201 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
202 ... ${dst['container_name']}
203 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
204 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
205 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
206 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
207 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
208 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
209 ... ${dst['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800210 Run Keyword and Ignore Error Collect Logs
211 END
212 Run Keyword and Ignore Error Collect Logs
213
suraj gour1ecfae92019-12-20 15:11:40 +0000214Check ONU adapter crash not forcing authentication again
215 [Documentation] After ONU adapter restart, checks wpa log for 'authentication started'
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700216 ... message count to make sure auth not started again and validates EAP status and ping.
suraj gour1ecfae92019-12-20 15:11:40 +0000217 ... Assuming that test1 or sanity was executed where all the ONUs are authenticated/DHCP/pingable
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800218 [Tags] functional ONUAdaptCrash notready
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700219 [Setup] None
220 [Teardown] None
suraj gour1ecfae92019-12-20 15:11:40 +0000221 @{before_list}= Create List
222 @{after_list}= Create List
223 FOR ${I} IN RANGE 0 ${num_onus}
224 ${src}= Set Variable ${hosts.src[${I}]}
225 ${dst}= Set Variable ${hosts.dst[${I}]}
226 ${onu_device_id}= Get Device ID From SN ${src['onu']}
227 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
228 ... ${of_id}
229 ${before}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
230 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
231 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700232 Append To List ${before_list} ${before}
suraj gour1ecfae92019-12-20 15:11:40 +0000233 END
234 Wait Until Keyword Succeeds ${timeout} 15s Restart Pod ${NAMESPACE} adapter-open-onu
235 Wait Until Keyword Succeeds ${timeout} 2s Validate Pod Status ${podName} ${NAMESPACE}
236 ... Running
237 FOR ${I} IN RANGE 0 ${num_onus}
238 ${src}= Set Variable ${hosts.src[${I}]}
239 ${dst}= Set Variable ${hosts.dst[${I}]}
240 ${onu_device_id}= Get Device ID From SN ${src['onu']}
241 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
242 ... ${of_id}
243 ${after}= Run Keyword If ${has_dataplane} Check Remote File Contents For WPA Logs
244 ... True /tmp/wpa.log authentication started ${src['ip']} ${src['user']} ${src['pass']}
245 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700246 Append To List ${after_list} ${after}
247 ${output}= Run Keyword If ${has_dataplane} Login And Run Command On Remote System
248 ... wpa_cli status | grep SUCCESS ${src['ip']} ${src['user']} ${src['pass']}
249 ... ${src['container_type']} ${src['container_name']}
Andy Bavier33e6dd32020-01-16 13:35:20 -0700250 Run Keyword If ${has_dataplane} Should Contain ${output} SUCCESS
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700251 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
252 ... Wait Until Keyword Succeeds 60s 2s Check Ping
suraj gour1ecfae92019-12-20 15:11:40 +0000253 ... True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']} ${src['ip']} ${src['user']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700254 ... ${src['pass']} ${src['container_type']} ${src['container_name']}
suraj gour1ecfae92019-12-20 15:11:40 +0000255 END
256 Lists Should Be Equal ${after_list} ${before_list}
257 Log ${after_list}
258 Log ${before_list}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700259 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800260
261Test Disable and Enable ONU scenario for ATT workflow
262 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
263 ... Assuming that test1 was executed where all the ONUs are authenticated/DHCP/pingable
264 ... Perform disable on the ONUs, call volt-remove-subscriber and validate that the pings do not succeed
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700265 ... Perform enable on the ONUs, authentication check, volt-add-subscriber-access and
266 ... validate that the pings are successful
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800267 ... VOL-2284
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800268 [Tags] functional ATT_DisableEnableONU
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800269 [Setup] None
270 #[Teardown] None
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800271 FOR ${I} IN RANGE 0 ${num_onus}
272 ${src}= Set Variable ${hosts.src[${I}]}
273 ${dst}= Set Variable ${hosts.dst[${I}]}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800274 ${onu_device_id}= Get Device ID From SN ${src['onu']}
275 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
276 ... ${of_id}
277 Disable Device ${onu_device_id}
Suchitra Vemuri04245f72020-01-08 16:43:24 -0800278 Sleep 5s
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800279 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
280 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700281 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
282 ... Wait Until Keyword Succeeds 60s 2s Check Ping
283 ... False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
284 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800285 ... ELSE sleep 60s
286 Enable Device ${onu_device_id}
287 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
288 ... ${ONOS_SSH_PORT} ${onu_port}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700289 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
290 ... Validate Authentication After Reassociate True
291 ... ${src['dp_iface_name']} ${src['ip']} ${src['user']} ${src['pass']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800292 ... ${src['container_type']} ${src['container_name']}
293 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700294 ... ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800295 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
296 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
297 Sleep 10s
298 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
Andy Bavier08ae5852019-12-19 09:12:42 -0700299 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800300 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
301 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
302 ... ${dst['container_name']}
303 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
304 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
Suchitra Vemuri5994cd12019-12-17 22:20:55 -0800305 Run Keyword and Ignore Error Collect Logs
suraj gourd5cfdbb2019-12-13 12:44:55 +0000306 END
307 Run Keyword and Ignore Error Collect Logs
308
Gayathri.Selvan49398962020-01-13 07:19:12 +0000309Verify restart any container after VOLTHA is operational
310 [Documentation] Restart any container after VOLTHA is operational.
311 ... Prerequisite : ONUs are authenticated and pingable.
Suchitra Vemurid1e2b712020-01-13 23:06:22 -0800312 [Tags] functional VOL-1958 RestartPods
Gayathri.Selvan49398962020-01-13 07:19:12 +0000313 [Setup] NONE
314 [Teardown] NONE
315 ${waitforRestart} Set Variable 120s
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700316 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000317 Log ${podStatusOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700318 ${countBforRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvan49398962020-01-13 07:19:12 +0000319 ${podName} Set Variable adapter-open-olt
320 Restart Pod ${NAMESPACE} ${podName}
321 Wait Until Keyword Succeeds ${waitforRestart} 2s Validate Pod Status ${podName} ${NAMESPACE}
322 ... Running
323 Repeat Sanity Test
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700324 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000325 Log ${podStatusOutput}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700326 ${countAfterRestart}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
Gayathri.Selvan49398962020-01-13 07:19:12 +0000327 Should Be Equal As Strings ${countAfterRestart} ${countBforRestart}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000328
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800329Delete OLT, ReAdd OLT and Perform Sanity Test
330 [Documentation] Validates E2E Ping Connectivity and object states for the given scenario:
331 ... Disable and Delete the OLT
332 ... Create/Enable the same OLT again
333 ... Validate authentication/DHCP/E2E pings succeed for all the ONUs connected to the OLT
Suchitra Vemuri2fa9bba2020-01-22 17:38:48 -0800334 [Tags] functional DeleteOLT
Suchitra Vemuri937cd392020-01-12 17:10:42 -0800335 [Setup] None
336 [Teardown] NONE
337 Run Keyword If ${has_dataplane} Clean Up Linux
338 Run Keyword If ${has_dataplane} Delete Device and Verify
339 Run Keyword and Ignore Error Collect Logs
340 # Recreate the OLT
341 Run Keyword If ${has_dataplane} Setup
342 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
343 Run Keyword and Ignore Error Collect Logs
344
Hemae6d223d2019-12-30 19:21:26 +0530345Adding the same OLT after enabling the device
346 [Documentation] Create OLT, enable it, Create the same OLT again and Check for the Error message
Suchitra Vemuri2fa9bba2020-01-22 17:38:48 -0800347 [Tags] VOL-2406 AddEnableOLT_AddTheSameOLTAgain notready
Hemae6d223d2019-12-30 19:21:26 +0530348 [Setup] Delete Device and Verify
349 [Teardown] None
350 Run Keyword If ${has_dataplane} Sleep 180s
Hemae6d223d2019-12-30 19:21:26 +0530351 ${olt_device_id}= Create Device ${olt_ip} ${OLT_PORT}
352 Set Suite Variable ${olt_device_id}
Hemae6d223d2019-12-30 19:21:26 +0530353 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device PREPROVISIONED UNKNOWN UNKNOWN
354 ... ${EMPTY} ${olt_device_id}
355 #Enable the created OLT device
356 Enable Device ${olt_device_id}
357 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE REACHABLE
358 ... ${olt_serial_number}
Hemae6d223d2019-12-30 19:21:26 +0530359 ${rc} ${output}= Run and Return Rc and Output
360 ... ${VOLTCTL_CONFIG}; voltctl device create -t openolt -H ${olt_ip}:${OLT_PORT}
361 Should Not Be Equal As Integers ${rc} 0
362 Log ${output}
Suchitra Vemuri1ef41262020-01-17 15:00:45 -0800363 Should Contain ${output} Device is already pre-provisioned
Hemae6d223d2019-12-30 19:21:26 +0530364 Log "This OLT is added already and enabled"
365
Hema783279b2020-01-22 15:37:37 +0530366Test disable ONUs and OLT then delete ONUs and OLT
367 [Documentation] On deployed POD, disable the ONU, disable the OLT and then delete ONU and OLT.
368 ... This TC is to confirm that ONU removal is not impacting OLT
369 ... Devices will be removed during the execution of this TC
370 ... so calling setup at the end to add the devices back to avoid the confusion.
371 [Tags] VOL-2354 DisableDeleteONUandOLT notready
372 [Setup] NONE
373 [Teardown] None
374 FOR ${I} IN RANGE 0 ${num_onus}
375 ${src}= Set Variable ${hosts.src[${I}]}
376 ${dst}= Set Variable ${hosts.dst[${I}]}
377 ${onu_device_id}= Get Device ID From SN ${src['onu']}
378 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE
379 ... REACHABLE ${src['onu']} onu=True onu_reason=omci-flows-pushed
380 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
381 ... REACHABLE ${olt_serial_number}
382 ${rc} ${output}= Run and Return Rc and Output
383 ... ${VOLTCTL_CONFIG}; voltctl device disable ${onu_device_id}
384 Should Be Equal As Integers ${rc} 0
385 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED UNKNOWN
386 ... REACHABLE ${src['onu']} onu=false
387 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device ENABLED ACTIVE
388 ... REACHABLE ${olt_serial_number}
389 END
390 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device disable ${olt_device_id}
391 Should Be Equal As Integers ${rc} 0
392 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN REACHABLE
393 ... ${olt_serial_number}
394 FOR ${I} IN RANGE 0 ${num_onus}
395 ${src}= Set Variable ${hosts.src[${I}]}
396 ${dst}= Set Variable ${hosts.dst[${I}]}
397 ${onu_device_id}= Get Device ID From SN ${src['onu']}
398 Wait Until Keyword Succeeds ${timeout} 5s Validate Device DISABLED DISCOVERED
399 ... UNREACHABLE ${src['onu']} onu=false
400 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${onu_device_id}
401 Should Be Equal As Integers ${rc} 0
402 Wait Until Keyword Succeeds ${timeout} 5s Validate OLT Device DISABLED UNKNOWN
403 ... REACHABLE ${olt_serial_number}
404 END
405 ${rc} ${output}= Run and Return Rc and Output ${VOLTCTL_CONFIG}; voltctl device delete ${olt_device_id}
406 Should Be Equal As Integers ${rc} 0
407 Wait Until Keyword Succeeds ${timeout} 5s Test Empty Device List
408 #Adding setup here to add the devices back since this TC removes the devices
409 Run Keyword If ${has_dataplane} sleep 180s
410 setup
411
David Bainbridgef81cd642019-11-20 00:14:47 +0000412Sanity E2E Test for OLT/ONU on POD With Core Fail and Restart
413 [Documentation] Deploys an device instance and waits for it to authenticate. After
414 ... authentication is successful the rw-core deployment is scaled to 0 instances to
415 ... simulate a POD crash. The test then scales the rw-core back to a single instance
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700416 ... and configures ONOS for access. The test succeeds if the device is able to
David Bainbridgef81cd642019-11-20 00:14:47 +0000417 ... complete the DHCP sequence.
418 [Tags] bbsim rwcore-restart
419 [Setup] Clear All Devices Then Create New Device
420 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
421 Set Global Variable ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000422 FOR ${I} IN RANGE 0 ${num_onus}
423 ${src}= Set Variable ${hosts.src[${I}]}
424 ${dst}= Set Variable ${hosts.dst[${I}]}
425 ${onu_device_id}= Get Device ID From SN ${src['onu']}
426 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
427 ... ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000428 # Bring up the device and verify it authenticates
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700429 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
David Bainbridgef81cd642019-11-20 00:14:47 +0000430 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed
431 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${k8s_node_ip}
432 ... ${ONOS_SSH_PORT} ${onu_port}
433 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
434 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
435 ... ${src['container_type']} ${src['container_name']}
436 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${k8s_node_ip}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700437 ... ${ONOS_SSH_PORT} ${onu_port}
David Bainbridgef81cd642019-11-20 00:14:47 +0000438 # Scale down the rw-core deployment to 0 PODs and once confirmed, scale it back to 1
439 Scale K8s Deployment voltha voltha-rw-core 0
440 Wait Until Keyword Succeeds ${timeout} 2s Pod Does Not Exist voltha voltha-rw-core
441 # Ensure the ofagent POD goes "not-ready" as expected
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700442 Wait Until keyword Succeeds ${timeout} 2s
443 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 0
David Bainbridgef81cd642019-11-20 00:14:47 +0000444 # Scale up the core deployment and make sure both it and the ofagent deployment are back
445 Scale K8s Deployment voltha voltha-rw-core 1
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700446 Wait Until Keyword Succeeds ${timeout} 2s
447 ... Check Expected Available Deployment Replicas voltha voltha-rw-core 1
448 Wait Until Keyword Succeeds ${timeout} 2s
449 ... Check Expected Available Deployment Replicas voltha voltha-ofagent 1
David Bainbridgef81cd642019-11-20 00:14:47 +0000450 # For some reason scaling down and up the POD behind a service causes the port forward to stop working,
451 # so restart the port forwarding for the API service
452 Restart VOLTHA Port Foward voltha-api-minimal
David Bainbridgef81cd642019-11-20 00:14:47 +0000453 # Ensure that the ofagent pod is up and ready and the device is available in ONOS, this
454 # represents system connectivity being restored
455 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
456 ... http://karaf:karaf@${k8s_node_ip}:${ONOS_REST_PORT} ${of_id}
David Bainbridgef81cd642019-11-20 00:14:47 +0000457 # Add subscriber access and verify that DHCP completes to ensure system is still functioning properly
458 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command ${k8s_node_ip}
459 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
460 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate DHCP and Ping True
461 ... True ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
462 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
463 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
464 ... ${dst['container_name']}
465 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
466 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
467 END
468
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700469*** Keywords ***
470Setup Suite
Andy Bavier88cd9f62019-11-26 16:22:33 -0700471 [Documentation] Set up the test suite
472 Common Test Suite Setup
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700473
David Bainbridgef81cd642019-11-20 00:14:47 +0000474Clear All Devices Then Create New Device
475 [Documentation] Remove any devices from VOLTHA and ONOS
David Bainbridgef81cd642019-11-20 00:14:47 +0000476 # Remove all devices from voltha and nos
477 Delete All Devices and Verify
David Bainbridgef81cd642019-11-20 00:14:47 +0000478 # Execute normal test Setup Keyword
479 Setup
480
Suchitra Vemuri582574b2019-12-10 21:11:54 -0800481Collect Logs
482 [Documentation] Collect Logs from voltha and onos cli for various commands
Suchitra Vemuri1a970a62019-11-26 12:52:16 -0800483 Run Keyword and Ignore Error Get Device List from Voltha
484 Run Keyword and Ignore Error Get Device Output from Voltha ${olt_device_id}
485 Run Keyword and Ignore Error Get Logical Device Output from Voltha ${logical_id}
Andy Bavierba9866b2019-10-11 07:11:53 -0700486 Run Keyword If ${external_libs} Get ONOS Status ${k8s_node_ip}
Andy Bavierba9866b2019-10-11 07:11:53 -0700487
Suchitra Vemuri65cd65f2019-08-30 14:39:22 -0700488Clean Up Linux
489 [Documentation] Kill processes and clean up interfaces on src+dst servers
Suchitra Vemuri8a9c3782019-10-23 12:43:01 -0700490 FOR ${I} IN RANGE 0 ${num_onus}
491 ${src}= Set Variable ${hosts.src[${I}]}
492 ${dst}= Set Variable ${hosts.dst[${I}]}
493 Run Keyword And Ignore Error Kill Linux Process [w]pa_supplicant ${src['ip']}
494 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
495 Run Keyword And Ignore Error Kill Linux Process [d]hclient ${src['ip']}
496 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
497 Run Keyword If '${dst['ip']}' != '${None}' Run Keyword And Ignore Error
498 ... Kill Linux Process [d]hcpd ${dst['ip']} ${dst['user']}
499 ... ${dst['pass']} ${dst['container_type']} ${dst['container_name']}
500 Delete IP Addresses from Interface on Remote Host ${src['dp_iface_name']} ${src['ip']}
501 ... ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
502 Run Keyword If '${dst['ip']}' != '${None}' Delete Interface on Remote Host
503 ... ${dst['dp_iface_name']}.${src['s_tag']} ${dst['ip']} ${dst['user']} ${dst['pass']}
504 ... ${dst['container_type']} ${dst['container_name']}
505 END
Andy Bavierba9866b2019-10-11 07:11:53 -0700506
Gayathri.Selvan49398962020-01-13 07:19:12 +0000507Repeat Sanity Test
Gayathri.Selvan9188ea52020-01-17 11:56:54 +0000508 [Documentation] Performs Sanity Test Procedure where sanity check is required with wpa reassociation
Gayathri.Selvan49398962020-01-13 07:19:12 +0000509 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${olt_serial_number}
510 Set Global Variable ${of_id}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000511 FOR ${I} IN RANGE 0 ${num_onus}
512 ${src}= Set Variable ${hosts.src[${I}]}
513 ${dst}= Set Variable ${hosts.dst[${I}]}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000514 ${onu_device_id}= Get Device ID From SN ${src['onu']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700515 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s
516 ... Get ONU Port in ONOS ${src['onu']} ${of_id}
517 Wait Until Keyword Succeeds ${timeout} 2s
518 ... Verify Eapol Flows Added For ONU ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
519 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
520 ... Validate Authentication After Reassociate
Gayathri.Selvan49398962020-01-13 07:19:12 +0000521 ... True ${src['dp_iface_name']} ${src['ip']} ${src['user']} ${src['pass']}
522 ... ${src['container_type']} ${src['container_name']}
Zack Williamsa8fe75a2020-01-10 14:25:27 -0700523 Wait Until Keyword Succeeds ${timeout} 2s
524 ... Verify ONU in AAA-Users ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
525 Wait Until Keyword Succeeds ${timeout} 2s
526 ... Execute ONOS CLI Command ${k8s_node_ip} ${ONOS_SSH_PORT}
527 ... volt-add-subscriber-access ${of_id} ${onu_port}
528 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
529 ... Validate DHCP and Ping True True
530 ... ${src['dp_iface_name']} ${src['s_tag']} ${src['c_tag']} ${dst['dp_iface_ip_qinq']}
Gayathri.Selvan49398962020-01-13 07:19:12 +0000531 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
532 ... ${dst['dp_iface_name']} ${dst['ip']} ${dst['user']} ${dst['pass']} ${dst['container_type']}
533 ... ${dst['container_name']}
534 Wait Until Keyword Succeeds ${timeout} 2s Run Keyword And Continue On Failure
535 ... Validate Subscriber DHCP Allocation ${k8s_node_ip} ${ONOS_SSH_PORT} ${onu_port}
536 Run Keyword and Ignore Error Get Device Output from Voltha ${onu_device_id}
537 Run Keyword and Ignore Error Collect Logs
Suchitra Vemuri15e15252020-01-20 17:38:49 -0800538 END