blob: 1058f0bc589ca1236936fc92c178388e6a5e5b7d [file] [log] [blame]
Hardik Windlassdd1a9a12021-02-23 15:34:50 +00001# Copyright 2021 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# FIXME Can we use the same test against BBSim and Hardware?
15
16*** Settings ***
17Documentation Test Voltha Components Software Upgrade
18Suite Setup Setup Suite
19Test Setup Setup
20Test Teardown Teardown
21Suite Teardown Teardown Suite
22Library Collections
23Library String
24Library OperatingSystem
25Library XML
26Library RequestsLibrary
27Library ../../libraries/DependencyLibrary.py
28Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../variables/variables.robot
34Resource ../../libraries/power_switch.robot
35
36*** Variables ***
37${POD_NAME} flex-ocp-cord
38${KUBERNETES_CONF} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
39${KUBERNETES_CONFIGS_DIR} ~/pod-configs/kubernetes-configs
40#${KUBERNETES_CONFIGS_DIR} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.conf
41${KUBERNETES_YAML} ${KUBERNETES_CONFIGS_DIR}/${POD_NAME}.yml
42${HELM_CHARTS_DIR} ~/helm-charts
43${VOLTHA_POD_NUM} 8
44${NAMESPACE} voltha
Hardik Windlass4288c6a2021-09-28 07:22:06 +000045${INFRA_NAMESPACE} default
Hardik Windlassdd1a9a12021-02-23 15:34:50 +000046# For below variable value, using deployment name as using grep for
47# parsing radius pod name, we can also use full radius pod name
48${RESTART_POD_NAME} radius
49${timeout} 60s
50${of_id} 0
51${logical_id} 0
52${has_dataplane} True
53${teardown_device} False
54${scripts} ../../scripts
55
56# Per-test logging on failure is turned off by default; set this variable to enable
57${container_log_dir} ${None}
58
Matteo Scandoloda1c64b2021-07-20 10:14:49 -070059# logging flag to enable Collect Logs, can be passed via the command line too
60# example: -v logging:False
61${logging} True
62
Hardik Windlassdd1a9a12021-02-23 15:34:50 +000063${suppressaddsubscriber} True
64
65# Voltha Components to Test for Software Upgrade need to be passed in the following variable in format:
Hardik Windlassdc2610f2021-03-09 07:33:51 +000066# <comp-label>,<comp-container>,<comp-image>*<comp-label>,<comp-container>,<comp-image>*
67# Example: adapter-open-olt,adapter-open-olt,voltha/voltha-openolt-adapter:3.1.3*
Hardik Windlassdd1a9a12021-02-23 15:34:50 +000068${voltha_comps_under_test} ${EMPTY}
69
70*** Test Cases ***
71Test Voltha Components Minor Version Upgrade
72 [Documentation] Validates the Voltha Components Minor Version Upgrade doesn't affect the system functionality
73 ... Performs the sanity and verifies all the ONUs are authenticated/DHCP/pingable
74 ... Requirement: Components to test needs to be passed in robot command variable 'voltha_comps_under_test' in the format:
Hardik Windlassdc2610f2021-03-09 07:33:51 +000075 ... <comp-label>,<comp-container>,<comp-image>*<comp-label>,<comp-container>,<comp-image>*
Hardik Windlassdd1a9a12021-02-23 15:34:50 +000076 ... Check [VOL-3843] for more details
77 [Tags] functional VolthaCompMinorVerUpgrade
78 [Setup] Start Logging VolthaCompMinorVerUpgrade
79 [Teardown] Run Keywords Collect Logs
80 ... AND Stop Logging VolthaCompMinorVerUpgrade
81 ... AND Delete All Devices and Verify
82 # Add OLT device
83 Setup
84 # Performing Sanity Test to make sure subscribers are all DHCP and pingable
85 Run Keyword If ${has_dataplane} Clean Up Linux
86 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
87 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
88 Log ${podStatusOutput}
89 ${countBeforeUpgrade}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
90 ${num_comps_under_test}= Get Length ${list_voltha_comps_under_test}
91 FOR ${I} IN RANGE 0 ${num_comps_under_test}
92 ${label}= Set Variable ${list_voltha_comps_under_test}[${I}][label]
93 ${container}= Set Variable ${list_voltha_comps_under_test}[${I}][container]
94 ${image}= Set Variable ${list_voltha_comps_under_test}[${I}][image]
Hardik Windlassdc2610f2021-03-09 07:33:51 +000095 ${pod_image} ${app_ver} ${helm_chart} Get Pod Image And App Version And Helm Chart By Label
96 ... ${NAMESPACE} app ${label}
97 Log ${label}: image, app ver & helm chart before upgrade: ${pod_image}, ${app_ver} & ${helm_chart}
Hardik Windlassdd1a9a12021-02-23 15:34:50 +000098 ${deployment}= Wait Until Keyword Succeeds ${timeout} 15s
99 ... Get K8s Deployment by Pod Label ${NAMESPACE} app ${label}
100 Wait Until Keyword Succeeds ${timeout} 15s Deploy Pod New Image ${NAMESPACE} ${deployment}
101 ... ${container} ${image}
102 Wait Until Keyword Succeeds ${timeout} 3s Validate Pods Status By Label ${NAMESPACE}
103 ... app ${label} Running
104 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${label}
105 Wait Until Keyword Succeeds ${timeout} 3s Verify Pod Image ${NAMESPACE} app ${label} ${image}
Hardik Windlassdc2610f2021-03-09 07:33:51 +0000106 ${pod_image_1} ${app_ver_1} ${helm_chart_1} Get Pod Image And App Version And Helm Chart By Label
107 ... ${NAMESPACE} app ${label}
108 Log ${label}: image, app ver & helm chart after upgrade: ${pod_image_1}, ${app_ver_1} & ${helm_chart_1}
Matteo Scandolo2769d2b2021-04-14 10:29:24 -0700109 Restart VOLTHA Port Forward voltha-api
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000110 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test ${suppressaddsubscriber}
111 END
112 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
113 Log ${podStatusOutput}
114 ${countAfterUpgrade}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
115 Should Be Equal As Strings ${countAfterUpgrade} ${countBeforeUpgrade}
116 # Additional Verification
117 Wait Until Keyword Succeeds ${timeout} 2s Delete All Devices and Verify
118 Setup
119 Run Keyword If ${has_dataplane} Clean Up Linux
120 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
121
Hardik Windlass07b52452022-01-17 07:35:53 +0000122
123Test Voltha Components Minor Version Rolling Upgrade
124 [Documentation] Validates that the system can handle operations during software upgrade for minor versions.
125 ... Requirement: Components to test needs to be passed in robot command variable 'voltha_comps_under_test' in the format:
126 ... <comp-label>,<comp-container>,<comp-image>*<comp-label>,<comp-container>,<comp-image>*
127 ... Check [VOL-4534] for more details
Hardik Windlass76f68212022-01-19 08:16:14 +0000128 [Tags] functional VolthaCompMinorVerRollingUpgrade
Hardik Windlass07b52452022-01-17 07:35:53 +0000129 [Setup] Start Logging VolthaCompMinorVerRollingUpgrade
130 [Teardown] Run Keywords Collect Logs
131 ... AND Stop Logging VolthaCompMinorVerRollingUpgrade
132 ... AND Delete All Devices and Verify
133 # Add OLT device
134 Setup
135 Run Keyword If ${has_dataplane} Clean Up Linux
136 FOR ${I} IN RANGE 0 ${olt_count}
137 ${olt_serial_number}= Get From Dictionary ${olt_ids}[${I}] sn
138 ${olt_device_id}= Get OLTDeviceID From OLT List ${olt_serial_number}
139 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS
140 ... ${olt_serial_number}
141 Wait Until Keyword Succeeds ${timeout} 2s Device Is Available In ONOS
142 ... http://karaf:karaf@${ONOS_REST_IP}:${ONOS_REST_PORT} ${of_id}
143 END
144 FOR ${I} IN RANGE 0 ${num_all_onus}
145 ${src}= Set Variable ${hosts.src[${I}]}
146 ${dst}= Set Variable ${hosts.dst[${I}]}
147 ${of_id}= Get ofID From OLT List ${src['olt']}
148 ${onu_device_id}= Get Device ID From SN ${src['onu']}
149 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
150 ... ${of_id} ${src['uni_id']}
151 # Check ONU port is Enabled in ONOS
152 Wait Until Keyword Succeeds 120s 2s
153 ... Verify UNI Port Is Enabled ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${src['onu']} ${src['uni_id']}
154 Wait Until Keyword Succeeds ${timeout} 5s
155 ... Verify Default Meter Present in ONOS ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
156 Wait Until Keyword Succeeds ${timeout} 2s Verify Eapol Flows Added For ONU ${ONOS_SSH_IP}
157 ... ${ONOS_SSH_PORT} ${of_id} ${onu_port}
158 # Bring up the device and verify it authenticates
159 Wait Until Keyword Succeeds ${timeout} 5s Validate Device ENABLED ACTIVE REACHABLE
160 ... ${onu_device_id} onu=True onu_reason=omci-flows-pushed by_dev_id=True
161 ${wpa_log}= Run Keyword If ${has_dataplane} Catenate SEPARATOR=.
162 ... /tmp/wpa ${src['dp_iface_name']} log
163 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure Validate Authentication True
164 ... ${src['dp_iface_name']} wpa_supplicant.conf ${src['ip']} ${src['user']} ${src['pass']}
165 ... ${src['container_type']} ${src['container_name']} ${wpa_log}
166 Wait Until Keyword Succeeds ${timeout} 2s Verify ONU in AAA-Users ${ONOS_SSH_IP}
167 ... ${ONOS_SSH_PORT} ${onu_port}
168 END
169 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
170 Log ${podStatusOutput}
171 ${countBeforeUpgrade}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
172 ${num_comps_under_test}= Get Length ${list_voltha_comps_under_test}
173 FOR ${I} IN RANGE 0 ${num_comps_under_test}
174 ${label}= Set Variable ${list_voltha_comps_under_test}[${I}][label]
175 ${container}= Set Variable ${list_voltha_comps_under_test}[${I}][container]
176 ${image}= Set Variable ${list_voltha_comps_under_test}[${I}][image]
177 ${pod_image} ${app_ver} ${helm_chart} Get Pod Image And App Version And Helm Chart By Label
178 ... ${NAMESPACE} app ${label}
179 Log ${label}: image, app ver & helm chart before upgrade: ${pod_image}, ${app_ver} & ${helm_chart}
180 ${deployment}= Wait Until Keyword Succeeds ${timeout} 15s
181 ... Get K8s Deployment by Pod Label ${NAMESPACE} app ${label}
182 Wait Until Keyword Succeeds ${timeout} 15s Deploy Pod New Image ${NAMESPACE} ${deployment}
183 ... ${container} ${image}
184 # Static sleep to let image-update progress before initiating subscriber-add system-operation
185 Sleep 2s
186 Provision Subscribers
187 Wait Until Keyword Succeeds ${timeout} 3s Validate Pods Status By Label ${NAMESPACE}
188 ... app ${label} Running
189 Wait Until Keyword Succeeds ${timeout} 3s Pods Are Ready By Label ${NAMESPACE} app ${label}
190 Wait Until Keyword Succeeds ${timeout} 3s Verify Pod Image ${NAMESPACE} app ${label} ${image}
191 ${pod_image_1} ${app_ver_1} ${helm_chart_1} Get Pod Image And App Version And Helm Chart By Label
192 ... ${NAMESPACE} app ${label}
193 Log ${label}: image, app ver & helm chart after upgrade: ${pod_image_1}, ${app_ver_1} & ${helm_chart_1}
194 Restart VOLTHA Port Forward voltha-api
Hardik Windlass515e9ee2022-02-24 11:52:31 +0000195 # Static sleep to let voltctl tcp connection establish
196 Sleep 5s
Hardik Windlass07b52452022-01-17 07:35:53 +0000197 Verify Provisioned Subscribers
198 Unprovision Subscribers
199 END
200 ${podStatusOutput}= Run kubectl get pods -n ${NAMESPACE}
201 Log ${podStatusOutput}
202 ${countAfterUpgrade}= Run kubectl get pods -n ${NAMESPACE} | grep Running | wc -l
203 Should Be Equal As Strings ${countAfterUpgrade} ${countBeforeUpgrade}
204 # Additional Verification
205 Wait Until Keyword Succeeds ${timeout} 2s Delete All Devices and Verify
206 Setup
207 Run Keyword If ${has_dataplane} Clean Up Linux
208 Wait Until Keyword Succeeds ${timeout} 2s Perform Sanity Test
209
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000210*** Keywords ***
211Setup Suite
212 [Documentation] Set up the test suite
213 Common Test Suite Setup
214 Create Voltha Comp Under Test List
215
216Teardown Suite
217 [Documentation] Tear down steps for the suite
218 Run Keyword If ${has_dataplane} Clean Up Linux
TorstenThieme4e2168e2021-06-22 14:01:47 +0000219 Close All ONOS SSH Connections
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000220
221Create Voltha Comp Under Test List
222 [Documentation] Creates a list of Voltha Components to Test from the input variable string
223 ... The input string is expected to be in format:
Hardik Windlassdc2610f2021-03-09 07:33:51 +0000224 ... <comp-label>,<comp-container>,<comp-image>*<comp-label>,<comp-container>,<comp-image>*
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000225 ${list_voltha_comps_under_test} Create List
226 @{comps_under_test_arr}= Split String ${voltha_comps_under_test} *
227 ${num_comps_under_test}= Get Length ${comps_under_test_arr}
Hardik Windlassdc2610f2021-03-09 07:33:51 +0000228 FOR ${I} IN RANGE 0 ${num_comps_under_test}-1
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000229 @{comp_under_test_arr}= Split String ${comps_under_test_arr[${I}]} ,
230 ${label}= Set Variable ${comp_under_test_arr[0]}
231 ${container}= Set Variable ${comp_under_test_arr[1]}
232 ${image}= Set Variable ${comp_under_test_arr[2]}
233 ${comp_under_test} Create Dictionary label ${label} container ${container} image ${image}
234 Append To List ${list_voltha_comps_under_test} ${comp_under_test}
235 END
Hardik Windlassdc2610f2021-03-09 07:33:51 +0000236 Log ${list_voltha_comps_under_test}
Hardik Windlassdd1a9a12021-02-23 15:34:50 +0000237 Set Suite Variable ${list_voltha_comps_under_test}
Hardik Windlass07b52452022-01-17 07:35:53 +0000238
239Provision Subscribers
240 [Documentation] This keyword provisions/adds all the subscribers on all the devices
241 FOR ${I} IN RANGE 0 ${num_all_onus}
242 ${src}= Set Variable ${hosts.src[${I}]}
243 ${dst}= Set Variable ${hosts.dst[${I}]}
244 ${of_id}= Get ofID From OLT List ${src['olt']}
245 ${onu_device_id}= Get Device ID From SN ${src['onu']}
246 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
247 ... ${of_id} ${src['uni_id']}
248 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
249 ... ${ONOS_SSH_PORT} volt-add-subscriber-access ${of_id} ${onu_port}
250 END
251
252Verify Provisioned Subscribers
253 [Documentation] This keyword verifies all the subscribers on all the devices
254 FOR ${I} IN RANGE 0 ${num_all_onus}
255 ${src}= Set Variable ${hosts.src[${I}]}
256 ${dst}= Set Variable ${hosts.dst[${I}]}
257 ${of_id}= Get ofID From OLT List ${src['olt']}
258 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
259 ${onu_device_id}= Get Device ID From SN ${src['onu']}
260 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
261 ... ${of_id} ${src['uni_id']}
262 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
263 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
264 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
265 ... Validate Subscriber DHCP Allocation ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
266 # Verify subscriber access flows are added for the ONU port
267 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 5s
268 ... Verify Subscriber Access Flows Added For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${of_id}
269 ... ${onu_port} ${nni_port} ${src['c_tag']} ${src['s_tag']}
270 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
271 ... Wait Until Keyword Succeeds ${timeout} 2s
272 ... Check Ping True ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
273 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
274 END
275
276Unprovision Subscribers
277 [Documentation] This keyword unprovisions/deletes all the subscribers on all the devices
278 FOR ${I} IN RANGE 0 ${num_all_onus}
279 ${src}= Set Variable ${hosts.src[${I}]}
280 ${dst}= Set Variable ${hosts.dst[${I}]}
281 ${of_id}= Get ofID From OLT List ${src['olt']}
282 ${nni_port}= Wait Until Keyword Succeeds ${timeout} 2s Get NNI Port in ONOS ${of_id}
283 ${onu_device_id}= Get Device ID From SN ${src['onu']}
284 ${onu_port}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${src['onu']}
285 ... ${of_id} ${src['uni_id']}
286 Wait Until Keyword Succeeds ${timeout} 2s Execute ONOS CLI Command use single connection ${ONOS_SSH_IP}
287 ... ${ONOS_SSH_PORT} volt-remove-subscriber-access ${of_id} ${onu_port}
288 Run Keyword And Continue On Failure Wait Until Keyword Succeeds ${timeout} 2s
289 ... Verify No Pending Flows For ONU ${ONOS_SSH_IP} ${ONOS_SSH_PORT} ${onu_port}
290 Run Keyword If ${has_dataplane} Run Keyword And Continue On Failure
291 ... Wait Until Keyword Succeeds ${timeout} 2s
292 ... Check Ping False ${dst['dp_iface_ip_qinq']} ${src['dp_iface_name']}
293 ... ${src['ip']} ${src['user']} ${src['pass']} ${src['container_type']} ${src['container_name']}
294 ... ELSE Sleep 15s
295 END