blob: 6177ce32683a0d5e8c869773bf8e98d0f1603f7c [file] [log] [blame]
TorstenThieme700ccbf2022-02-11 10:24:05 +00001# Copyright 2022 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15*** Settings ***
16Documentation Test of try to catch memory leak in voltha components.
17Suite Setup Setup Suite
18Suite Teardown Teardown Suite
19Test Setup Setup
20Test Teardown Teardown
21Library Collections
22Library String
23Library OperatingSystem
24Library XML
25Library RequestsLibrary
26Library ../../libraries/DependencyLibrary.py
TorstenThieme7e8ccb92022-09-29 09:12:38 +000027Library ../../libraries/utility.py WITH NAME utility
TorstenThieme700ccbf2022-02-11 10:24:05 +000028Resource ../../libraries/onos.robot
29Resource ../../libraries/voltctl.robot
30Resource ../../libraries/voltha.robot
31Resource ../../libraries/utils.robot
32Resource ../../libraries/k8s.robot
33Resource ../../libraries/onu_utilities.robot
34Resource ../../variables/variables.robot
35
36*** Variables ***
37${NAMESPACE} voltha
38${INFRA_NAMESPACE} default
39${timeout} 60s
40${of_id} 0
41${logical_id} 0
42${has_dataplane} True
43${external_libs} True
44${teardown_device} True
45${scripts} ../../scripts
46# Per-test logging on failure is turned off by default; set this variable to enable
47${container_log_dir} ${None}
48# determines the environment workflow: DT, TT or ATT (default)
49# example: -v workflow:DT
50${workflow} ATT
51# KV Store Prefix
52# example: -v kvstoreprefix:voltha/voltha_voltha
53${kvstoreprefix} voltha/voltha_voltha
54# flag debugmode is used, if true timeout calculation various, can be passed via the command line too
55# example: -v debugmode:True
56${debugmode} False
57# logging flag to enable Collect Logs, can be passed via the command line too
58# example: -v logging:True
59${logging} False
60# if True execution will be paused before clean up, only use in case of manual testing, do not use in ci pipeline!
61# example: -v pausebeforecleanup:True
62${pausebeforecleanup} False
63# if True some outputs to console are done during running tests e.g. long duration flow test
64# example: -v print2console:True
65${print2console} False
66# if True etcd check will be executed in test case teardown, if False etcd check will be executed in suite teardown
67# example: -v etcdcheckintestteardown:False
68${etcdcheckintestteardown} True
69${data_dir} ../data
70# number of iterations
71# example: -v iterations:10
TorstenThieme7e8ccb92022-09-29 09:12:38 +000072${iterations} 50
73# address of Prometheus
74# example: -v prometheusaddr:0.0.0.0
75${prometheusaddr} 0.0.0.0
76# port of Prometheus
77# example: -v prometheusport:31301
78${prometheusport} 31301
TorstenThieme700ccbf2022-02-11 10:24:05 +000079
80# flag to choose the subscriber provisioning command type in ONOS
81# TT often provision a single services for a subscriber (eg: hsia, voip, ...) one after the other.
82# if set to True, command used is "volt-add-subscriber-unitag"
83# if set to False, comand used is "volt-add-subscriber-access"
84${unitag_sub} False
85
86*** Test Cases ***
87Memory Leak Test Openonu Go Adapter
88 [Documentation] Test of try to catch memory leak in Openonu Go Adapter for all three workflows, ATT, DT and TT
89 ... Multiple run of Flow and ONU setup and teardown to try to catch memory leak.
90 ... - do workflow related sanity test (bring up onu to omci flows pushed and setup flows)
91 ... - remove flows
92 ... - delete ONU devices
93 ... - wait for onu auto detect
TorstenThieme7e8ccb92022-09-29 09:12:38 +000094 ... Attention: Due VOL-4703 is not corrected memory leak tests will run in pipeline for DT workflow only!
TorstenThieme2445db22022-09-09 11:23:47 +000095 ... This is a temporaly workaround only! Has to be checked after introduction of voltha-go-controller.
TorstenThieme700ccbf2022-02-11 10:24:05 +000096 [Tags] functionalMemoryLeak MemoryLeakTestOnuGo
97 [Setup] Run Keywords Start Logging MemoryLeakTestOnuGo
98 ... AND Setup
99 Run Keyword If ${has_dataplane} Clean Up Linux
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000100 ${output_file}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptionsOpenOnuAdapter.txt
101 Create File ${output_file} This file contains the memory consumptions of openonu adapter.
TorstenThieme700ccbf2022-02-11 10:24:05 +0000102 Run Keyword If ${print2console} Log \r\nStart ${iterations} iterations. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000103 ${start_mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} adapter-open-onu voltha
104 Run Keyword If ${print2console} Log \r\nMemory consumptions Start ${start_mem_consumption} console=yes
105 ${time}= Get Time
106 Append To File ${output_file} \r\nMemory consumptions Start: ${start_mem_consumption} at ${time}.
TorstenThieme700ccbf2022-02-11 10:24:05 +0000107 FOR ${I} IN RANGE 1 ${iterations} + 1
108 Run Keyword If ${print2console} Log \r\nStart iteration ${I} of ${iterations}. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000109 Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
110 ... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
111 ... ELSE Perform Sanity Test
TorstenThieme1208ab52022-04-11 14:24:27 +0000112 Sleep 5s
TorstenThieme700ccbf2022-02-11 10:24:05 +0000113 Run Keyword If ${print2console} Log Remove Flows. console=yes
114 Remove Flows all ONUs
115 Run Keyword If ${print2console} Log Check Flows removed. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000116 Check All Flows Removed
TorstenThieme1208ab52022-04-11 14:24:27 +0000117 Run Keyword If ${print2console} Log Get ONU Device IDs. console=yes
118 ${onu_device_id_list}= Get ONUs Device IDs from Voltha
TorstenThieme700ccbf2022-02-11 10:24:05 +0000119 Run Keyword If ${print2console} Log Delete ONUs. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000120 Wait Until Keyword Succeeds ${timeout} 1s Delete Devices In Voltha Type=brcm_openomci_onu
TorstenThieme700ccbf2022-02-11 10:24:05 +0000121 Run Keyword If ${print2console} Log Wait for ONUs come back. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000122 Wait Until Keyword Succeeds ${timeout} 1s Check for new ONU Device IDs ${onu_device_id_list}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000123 ${list_onus} Create List
124 Build ONU SN List ${list_onus}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000125 Wait Until Keyword Succeeds ${timeout} 1s Check all ONU OperStatus ${list_onus} ACTIVE
TorstenThieme1208ab52022-04-11 14:24:27 +0000126 Build ONU SN List ${list_onus}
127 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
128 ... "${workflow}"=="TT" initial-mib-downloaded
129 ... "${workflow}"=="ATT" omci-flows-pushed
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000130 Wait Until Keyword Succeeds ${timeout} 1s
TorstenThieme2445db22022-09-09 11:23:47 +0000131 ... Validate ONU Devices ENABLED ACTIVE REACHABLE ${list_onus} onu_reason=${onu_reason}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000132 ${mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} adapter-open-onu voltha
133 Run Keyword If ${print2console} Log \r\nMemory consumptions ${I} ${mem_consumption} console=yes
TorstenThieme700ccbf2022-02-11 10:24:05 +0000134 Run Keyword If ${print2console} Log End iteration ${I} of ${iterations}. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000135 ${time}= Get Time
136 Append To File ${output_file} \r\nMemory consumptions ${I}: ${mem_consumption} at ${time}.
TorstenThieme700ccbf2022-02-11 10:24:05 +0000137 END
138 [Teardown] Run Keywords Printout ONU Serial Number and Device Id print2console=${print2console}
139 ... AND Run Keyword If ${logging} Get Logical Id of OLT
140 ... AND Run Keyword If ${logging} Collect Logs
141 ... AND Teardown Test
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000142 ... AND Run Keyword And Ignore Error Wait Until Keyword Succeeds 300 10s
143 ... Validate Memory Consumptions adapter-open-onu voltha ${start_mem_consumption} ${output_file}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000144 ... AND Stop Logging MemoryLeakTestOnuGo
145
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000146Memory Leak Test Openolt Adapter
147 [Documentation] Test of try to catch memory leak in Openolt Adapter for all three workflows, ATT, DT and TT
148 ... Multiple run of OLT setup and teardown to try to catch memory leak.
149 ... - do workflow related sanity test (bring up onu to omci flows pushed and setup flows)
150 ... - delete OLT devices
151 ... - wait for OLT is removed
152 ... - add and enable OLT again
153 ... - wait for ONUs available again
154 ... Hint: default timeout in BBSim to mimic OLT reboot is 60 seconds!
155 ... This behaviour of BBSim can be modified by 'oltRebootDelay: 60' in BBSim section of helm chart or
156 ... used values.yaml during 'voltha up'.
157 ... Attention: Due VOL-4703 is not corrected memory leak tests will run in pipeline for DT workflow only!
158 ... This is a temporaly workaround only! Has to be checked after introduction of voltha-go-controller.
159 [Tags] functionalMemoryLeak MemoryLeakTestOlt
160 [Setup] Run Keywords Start Logging MemoryLeakTestOlt
161 ... AND Setup
162 Run Keyword If ${has_dataplane} Clean Up Linux
163 ${output_file}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptionsOpenOltAdapter.txt
164 Create File ${output_file} This file contains the memory consumptions of openolt adapter.
165 Run Keyword If ${print2console} Log \r\nStart ${iterations} iterations. console=yes
166 ${start_mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} adapter-open-olt voltha
167 Run Keyword If ${print2console} Log \r\nMemory consumptions Start ${start_mem_consumption} console=yes
168 ${time}= Get Time
169 Append To File ${output_file} \r\nMemory consumptions Start: ${start_mem_consumption} at ${time}.
170 FOR ${I} IN RANGE 1 ${iterations} + 1
171 Run Keyword If ${print2console} Log \r\nStart iteration ${I} of ${iterations}. console=yes
172 Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
173 ... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
174 ... ELSE Perform Sanity Test
175 Sleep 5s
176 Run Keyword If ${print2console} Log Delete OLTs. console=yes
177 Delete Devices In Voltha Type=openolt
178 Run Keyword If ${print2console} Log Check OLTs removed. console=yes
179 Wait Until Keyword Succeeds ${timeout} 1s Test Empty Device List
180 Sleep 20s
181 Run Keyword If ${print2console} Log Add OLTs (calling Setup). console=yes
182 Setup
183 Run Keyword If ${print2console} Log Wait for ONUs come back. console=yes
184 ${list_onus} Create List
185 Build ONU SN List ${list_onus}
186 Wait Until Keyword Succeeds ${timeout} 1s Check all ONU OperStatus ${list_onus} ACTIVE
187 Build ONU SN List ${list_onus}
188 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
189 ... "${workflow}"=="TT" initial-mib-downloaded
190 ... "${workflow}"=="ATT" omci-flows-pushed
191 Wait Until Keyword Succeeds ${timeout} 1s
192 ... Validate ONU Devices ENABLED ACTIVE REACHABLE ${list_onus} onu_reason=${onu_reason}
193 ${mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} adapter-open-olt voltha
194 Run Keyword If ${print2console} Log \r\nMemory consumptions ${I} ${mem_consumption} console=yes
195 Run Keyword If ${print2console} Log End iteration ${I} of ${iterations}. console=yes
196 ${time}= Get Time
197 Append To File ${output_file} \r\nMemory consumptions ${I}: ${mem_consumption} at ${time}.
198 END
199 [Teardown] Run Keywords Printout ONU Serial Number and Device Id print2console=${print2console}
200 ... AND Run Keyword If ${logging} Get Logical Id of OLT
201 ... AND Run Keyword If ${logging} Collect Logs
202 ... AND Teardown Test
203 ... AND Run Keyword And Ignore Error Wait Until Keyword Succeeds 300 10s
204 ... Validate Memory Consumptions adapter-open-olt voltha ${start_mem_consumption} ${output_file}
205 ... AND Stop Logging MemoryLeakTestOlt
206
207
TorstenThieme700ccbf2022-02-11 10:24:05 +0000208*** Keywords ***
209Setup Suite
210 [Documentation] Set up the test suite
211 Start Logging Setup or Teardown Setup-${SUITE NAME}
212 ${LogInfo}= Catenate
213 ... \r\nPassed arguments:
214 ... debugmode:${debugmode}, logging:${logging}, pausebeforecleanup:${pausebeforecleanup},
215 ... print2console:${print2console}, workflow:${workflow}, kvstoreprefix:${kvstoreprefix},
216 ... iterations:${iterations}
217 Log ${LogInfo} console=yes
218 Common Test Suite Setup
219 # set tech profiles
220 ${preload_tech_profile}= Set Variable If ${unitag_sub} and "${workflow}"=="TT" and not ${has_dataplane} True False
221 Set Suite Variable ${preload_tech_profile}
222 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-HSIA ${INFRA_NAMESPACE} 64
223 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-VoIP ${INFRA_NAMESPACE} 65
224 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-multi-uni-MCAST-AdditionalBW-None ${INFRA_NAMESPACE} 66
225 # delete etcd MIB Template Data
226 Delete MIB Template Data ${INFRA_NAMESPACE}
227 # delete etcd onu data
228 Delete ONU Go Adapter ETCD Data namespace=${INFRA_NAMESPACE} validate=True
229 Run Keyword If ${logging} Collect Logs
230 Stop Logging Setup or Teardown Setup-${SUITE NAME}
231
232
233Teardown Suite
234 [Documentation] Replaces the Suite Teardown in utils.robot.
235 ... Cleans up and checks all ONU ports disabled in ONOS.
236 ... Furthermore gives the possibility to pause the execution.
237 Start Logging Setup or Teardown Teardown-${SUITE NAME}
238 Run Keyword If ${pausebeforecleanup} Import Library Dialogs
239 Run Keyword If ${pausebeforecleanup} Pause Execution Press OK to continue with clean up!
240 Run Keyword If ${pausebeforecleanup} Log Teardown will be continued... console=yes
241 Run Keyword If ${teardown_device} Delete All Devices and Verify
242 Run Keyword Unless ${etcdcheckintestteardown} Wait Until Keyword Succeeds ${timeout} 1s
243 ... Validate Onu Data In Etcd ${INFRA_NAMESPACE} 0 ${kvstoreprefix} without_pm_data=False
244 Wait for Ports in ONOS for all OLTs ${ONOS_SSH_IP} ${ONOS_SSH_PORT} 0 BBSM ${timeout}
245 Run Keyword If ${logging} Collect Logs
246 Stop Logging Setup or Teardown Teardown-${SUITE NAME}
247 Close All ONOS SSH Connections
248 Set Suite Variable ${TechProfile} ${EMPTY}
249 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 64
250 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 65
251 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 66
252
253Teardown Test
254 [Documentation] Post-test Teardown
TorstenThieme22491b72022-03-31 13:12:27 +0000255 # log ONOS flows after remove check
256 ${flow}= Run Keyword If "${TEST STATUS}"=="FAIL" Execute ONOS CLI Command use single connection
257 ... ${ONOS_SSH_IP} ${ONOS_SSH_PORT} flows -s any ${of_id}
258 Run Keyword If "${TEST STATUS}"=="FAIL" Log ${flow}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000259 Run Keyword If ${pausebeforecleanup} Import Library Dialogs
260 Run Keyword If ${pausebeforecleanup} Pause Execution Press OK to continue with clean up!
261 Run Keyword If ${pausebeforecleanup} Log Teardown will be continued... console=yes
262 Run Keyword If ${teardown_device} Delete All Devices and Verify
263 # delete etcd MIB Template Data
264 Delete MIB Template Data ${INFRA_NAMESPACE}
265 # check etcd data are empty
266 Run Keyword If ${etcdcheckintestteardown} Wait Until Keyword Succeeds ${timeout} 1s
267 ... Validate Onu Data In Etcd ${INFRA_NAMESPACE} 0 ${kvstoreprefix} without_pm_data=False
268 Sleep 5s
TorstenThieme1208ab52022-04-11 14:24:27 +0000269
270Check for new ONU Device IDs
271 [Documentation] Checks that no old onu device ids stay
272 [Arguments] ${old_device_ids}
273 ${new_device_ids}= Get ONUs Device IDs from Voltha
274 Should Not Be Empty ${new_device_ids} No new ONU device IDs
275 FOR ${item} IN @{old_device_ids}
276 List Should Not Contain Value ${new_device_ids} ${item} Old device id ${item} still present.
277 END
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000278
279Validate Memory Consumptions
280 [Documentation] Validates meory consumptions of passed POD
281 [Arguments] ${container} ${namespace} ${start_value} ${output_file}
282 ${mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} ${container} ${namespace}
283 Run Keyword If ${print2console} Log \r\nMemory consumptions Teardwown ${mem_consumption} console=yes
284 ${time}= Get Time
285 Append To File ${output_file} \r\nMemory consumptions Teardown: ${mem_consumption} at ${time}.
286 ${upper_bound}= Evaluate (${start_value} + (${start_value}*0.3))
287 Should Be True ${upper_bound} >= ${mem_consumption}