blob: 6251ad19c5b14583e2202591f06d6de54085ebb8 [file] [log] [blame]
TorstenThieme700ccbf2022-02-11 10:24:05 +00001# Copyright 2022 - present Open Networking Foundation
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
TorstenThieme0fa54432023-02-14 13:27:20 +000015# Use bbsim-kind-dt-1OLTx1PONx2ONU.yaml
16
TorstenThieme700ccbf2022-02-11 10:24:05 +000017*** Settings ***
18Documentation Test of try to catch memory leak in voltha components.
19Suite Setup Setup Suite
20Suite Teardown Teardown Suite
21Test Setup Setup
22Test Teardown Teardown
23Library Collections
24Library String
25Library OperatingSystem
26Library XML
27Library RequestsLibrary
28Library ../../libraries/DependencyLibrary.py
TorstenThieme7e8ccb92022-09-29 09:12:38 +000029Library ../../libraries/utility.py WITH NAME utility
TorstenThieme700ccbf2022-02-11 10:24:05 +000030Resource ../../libraries/onos.robot
31Resource ../../libraries/voltctl.robot
32Resource ../../libraries/voltha.robot
33Resource ../../libraries/utils.robot
34Resource ../../libraries/k8s.robot
35Resource ../../libraries/onu_utilities.robot
36Resource ../../variables/variables.robot
37
38*** Variables ***
39${NAMESPACE} voltha
40${INFRA_NAMESPACE} default
41${timeout} 60s
42${of_id} 0
43${logical_id} 0
44${has_dataplane} True
45${external_libs} True
46${teardown_device} True
47${scripts} ../../scripts
48# Per-test logging on failure is turned off by default; set this variable to enable
49${container_log_dir} ${None}
50# determines the environment workflow: DT, TT or ATT (default)
51# example: -v workflow:DT
52${workflow} ATT
53# KV Store Prefix
54# example: -v kvstoreprefix:voltha/voltha_voltha
55${kvstoreprefix} voltha/voltha_voltha
56# flag debugmode is used, if true timeout calculation various, can be passed via the command line too
57# example: -v debugmode:True
58${debugmode} False
59# logging flag to enable Collect Logs, can be passed via the command line too
60# example: -v logging:True
61${logging} False
62# if True execution will be paused before clean up, only use in case of manual testing, do not use in ci pipeline!
63# example: -v pausebeforecleanup:True
64${pausebeforecleanup} False
65# if True some outputs to console are done during running tests e.g. long duration flow test
66# example: -v print2console:True
67${print2console} False
68# if True etcd check will be executed in test case teardown, if False etcd check will be executed in suite teardown
69# example: -v etcdcheckintestteardown:False
70${etcdcheckintestteardown} True
71${data_dir} ../data
72# number of iterations
73# example: -v iterations:10
TorstenThieme7e8ccb92022-09-29 09:12:38 +000074${iterations} 50
75# address of Prometheus
76# example: -v prometheusaddr:0.0.0.0
77${prometheusaddr} 0.0.0.0
78# port of Prometheus
79# example: -v prometheusport:31301
80${prometheusport} 31301
TorstenThieme700ccbf2022-02-11 10:24:05 +000081
82# flag to choose the subscriber provisioning command type in ONOS
83# TT often provision a single services for a subscriber (eg: hsia, voip, ...) one after the other.
84# if set to True, command used is "volt-add-subscriber-unitag"
85# if set to False, comand used is "volt-add-subscriber-access"
86${unitag_sub} False
87
88*** Test Cases ***
89Memory Leak Test Openonu Go Adapter
90 [Documentation] Test of try to catch memory leak in Openonu Go Adapter for all three workflows, ATT, DT and TT
91 ... Multiple run of Flow and ONU setup and teardown to try to catch memory leak.
TorstenThieme0fa54432023-02-14 13:27:20 +000092 ... Setup OLT and one (first) ONU, both will kept over the whole test
93 ... Setup a second ONU and do following in specified loops (iterations):
TorstenThieme700ccbf2022-02-11 10:24:05 +000094 ... - do workflow related sanity test (bring up onu to omci flows pushed and setup flows)
95 ... - remove flows
TorstenThieme0fa54432023-02-14 13:27:20 +000096 ... - delete ONU
TorstenThieme700ccbf2022-02-11 10:24:05 +000097 ... - wait for onu auto detect
TorstenThieme7e8ccb92022-09-29 09:12:38 +000098 ... Attention: Due VOL-4703 is not corrected memory leak tests will run in pipeline for DT workflow only!
TorstenThieme2445db22022-09-09 11:23:47 +000099 ... This is a temporaly workaround only! Has to be checked after introduction of voltha-go-controller.
TorstenThieme700ccbf2022-02-11 10:24:05 +0000100 [Tags] functionalMemoryLeak MemoryLeakTestOnuGo
101 [Setup] Run Keywords Start Logging MemoryLeakTestOnuGo
TorstenThieme0fa54432023-02-14 13:27:20 +0000102 ... AND Append Memory Consumption To File isTest=True action=test_start
TorstenThieme700ccbf2022-02-11 10:24:05 +0000103 Run Keyword If ${has_dataplane} Clean Up Linux
TorstenThieme0fa54432023-02-14 13:27:20 +0000104 ${output_file_onu}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptionsOpenOnuAdapterOnuTest.txt
105 Create File ${output_file_onu} This file contains the memory consumptions of openonu adapter.
106 Append To File ${output_file_onu} \r\nTest: ${TEST NAME}
107 ${start_mem_consumption_onu}= Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE}
108 ... ${output_file_onu} Start settling_memory=True
109 ${Device_Setup}= Set Variable True
110 Set Global Variable ${Device_Setup}
111 Setup
112 Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE}
113 ... ${output_file_onu} Setup-OLT #settling_memory=True
114 Append Memory Consumption To File isTest=True action=test_setup
115 # Start first ONU
116 ${src_onu_1}= Set Variable ${hosts.src[${0}]}
117 ${bbsim_pod}= Get Pod Name By Label ${NAMESPACE} release bbsim0
118 Power On ONU ${NAMESPACE} ${bbsim_pod} ${src_onu_1['onu']}
119 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
120 ... "${workflow}"=="TT" initial-mib-downloaded
121 ... "${workflow}"=="ATT" omci-flows-pushed
122 ${admin_state} ${oper_status} ${connect_status} ${onu_state_nb} ${onu_state}=
123 ... Map State ${onu_reason}
124 Wait Until Keyword Succeeds ${timeout} 1s
125 ... Validate Device ${admin_state} ${oper_status} ${connect_status}
126 ... ${src_onu_1['onu']} onu=True onu_reason=${onu_state}
127 ${setup_mem_consumption_onu}= Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE}
128 ... ${output_file_onu} Setup-ONU-1 settling_memory=True
129 # Start second ONU
130 ${src_onu_2}= Set Variable ${hosts.src[${1}]}
131 ${bbsim_pod}= Get Pod Name By Label ${NAMESPACE} release bbsim0
132 Power On ONU ${NAMESPACE} ${bbsim_pod} ${src_onu_2['onu']}
133 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
134 ... "${workflow}"=="TT" initial-mib-downloaded
135 ... "${workflow}"=="ATT" omci-flows-pushed
136 ${admin_state} ${oper_status} ${connect_status} ${onu_state_nb} ${onu_state}=
137 ... Map State ${onu_reason}
138 Wait Until Keyword Succeeds ${timeout} 1s
139 ... Validate Device ${admin_state} ${oper_status} ${connect_status}
140 ... ${src_onu_2['onu']} onu=True onu_reason=${onu_state}
141 Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE} ${output_file_onu} Setup-ONU-2
142 Append Memory Consumption To File isTest=True action=test_setup_onu2
TorstenThieme700ccbf2022-02-11 10:24:05 +0000143 FOR ${I} IN RANGE 1 ${iterations} + 1
144 Run Keyword If ${print2console} Log \r\nStart iteration ${I} of ${iterations}. console=yes
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000145 Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
146 ... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
147 ... ELSE Perform Sanity Test
TorstenThieme1208ab52022-04-11 14:24:27 +0000148 Sleep 5s
TorstenThieme700ccbf2022-02-11 10:24:05 +0000149 Run Keyword If ${print2console} Log Remove Flows. console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000150 # Remove Flows
151 ${src}= Set Variable ${hosts.src[1]}
152 ${of_id}= Wait Until Keyword Succeeds ${timeout} 15s Validate OLT Device in ONOS ${src_onu_2['olt']}
153 ${onu_sn_2}= Set Variable ${src_onu_2['onu']}
154 ${onu_port_2}= Wait Until Keyword Succeeds ${timeout} 2s Get ONU Port in ONOS ${onu_sn_2}
155 ... ${of_id} ${src_onu_2['uni_id']}
156 Remove Flows Conditional ${unitag_sub} ${onu_sn_2} ${of_id} ${onu_port_2}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000157 Run Keyword If ${print2console} Log Check Flows removed. console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000158 # Check All Flows Removed
159 ${expected_flows_onu}= Set Variable If "${workflow}"=="ATT" 1 0
160 Wait Until Keyword Succeeds ${timeout} 2s Validate number of flows ${ONOS_SSH_IP} ${ONOS_SSH_PORT}
161 ... ${expected_flows_onu} ${of_id} any ${onu_port_2}
162 # Delete Device
TorstenThieme1208ab52022-04-11 14:24:27 +0000163 Run Keyword If ${print2console} Log Get ONU Device IDs. console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000164 ${onu_device_id}= Get Device ID From SN ${onu_sn_2}
165 Run Keyword If ${print2console} Log Delete ONU. (device id: ${onu_device_id}) console=yes
166 Wait Until Keyword Succeeds ${timeout} 1s Delete Device ${onu_device_id}
167 Run Keyword If ${print2console} Log Wait for ONU come back. console=yes
168 ${onu_device_id_list}= Create List ${onu_device_id}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000169 Wait Until Keyword Succeeds ${timeout} 1s Check for new ONU Device IDs ${onu_device_id_list}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000170 ${list_onus} Create List
171 Build ONU SN List ${list_onus}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000172 Wait Until Keyword Succeeds ${timeout} 1s Check all ONU OperStatus ${list_onus} ACTIVE
TorstenThieme1208ab52022-04-11 14:24:27 +0000173 Build ONU SN List ${list_onus}
174 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
175 ... "${workflow}"=="TT" initial-mib-downloaded
176 ... "${workflow}"=="ATT" omci-flows-pushed
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000177 Wait Until Keyword Succeeds ${timeout} 1s
TorstenThieme0fa54432023-02-14 13:27:20 +0000178 ... Validate Device ENABLED ACTIVE REACHABLE ${onu_sn_2} onu_reason=${onu_reason} onu=True
179 ${formatedIt}= Format String {:>3} ${I}
180 Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE} ${output_file_onu}
181 ... Iteration ${formatedIt}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000182 END
TorstenThieme0fa54432023-02-14 13:27:20 +0000183 # Switch off second ONU
184 Power Off ONU ${NAMESPACE} ${bbsim_pod} ${src_onu_2['onu']}
TorstenThieme30c64752023-03-01 10:24:58 +0000185 Wait Until Keyword Succeeds 500s 30s
TorstenThieme0fa54432023-02-14 13:27:20 +0000186 ... Validate Memory Consumptions adapter-open-onu ${NAMESPACE} ${setup_mem_consumption_onu} ${output_file_onu}
187 ... out_string=Validate
TorstenThieme700ccbf2022-02-11 10:24:05 +0000188 [Teardown] Run Keywords Printout ONU Serial Number and Device Id print2console=${print2console}
189 ... AND Run Keyword If ${logging} Get Logical Id of OLT
190 ... AND Run Keyword If ${logging} Collect Logs
191 ... AND Teardown Test
TorstenThieme0fa54432023-02-14 13:27:20 +0000192 ... AND Append Memory Consumption To File isTest=True compare_to=test_setup
TorstenThieme30c64752023-03-01 10:24:58 +0000193 ... AND Run Keyword And Ignore Error Wait Until Keyword Succeeds 300s 30s
TorstenThieme0fa54432023-02-14 13:27:20 +0000194 ... Validate Memory Consumptions adapter-open-onu ${NAMESPACE} ${start_mem_consumption_onu} ${output_file_onu}
195 ... AND Append Memory Consumption To File isTest=True compare_to=test_start
TorstenThieme700ccbf2022-02-11 10:24:05 +0000196 ... AND Stop Logging MemoryLeakTestOnuGo
197
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000198Memory Leak Test Openolt Adapter
199 [Documentation] Test of try to catch memory leak in Openolt Adapter for all three workflows, ATT, DT and TT
200 ... Multiple run of OLT setup and teardown to try to catch memory leak.
201 ... - do workflow related sanity test (bring up onu to omci flows pushed and setup flows)
202 ... - delete OLT devices
203 ... - wait for OLT is removed
204 ... - add and enable OLT again
205 ... - wait for ONUs available again
TorstenThieme0fa54432023-02-14 13:27:20 +0000206 ... Hint: Also memory consumptions of Openonu GO Adapter will be validated!
207 ... Hint: default timePower On ONU Device ${NAMESPACE}out in BBSim to mimic OLT reboot is 60 seconds!
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000208 ... This behaviour of BBSim can be modified by 'oltRebootDelay: 60' in BBSim section of helm chart or
209 ... used values.yaml during 'voltha up'.
210 ... Attention: Due VOL-4703 is not corrected memory leak tests will run in pipeline for DT workflow only!
211 ... This is a temporaly workaround only! Has to be checked after introduction of voltha-go-controller.
212 [Tags] functionalMemoryLeak MemoryLeakTestOlt
213 [Setup] Run Keywords Start Logging MemoryLeakTestOlt
TorstenThieme0fa54432023-02-14 13:27:20 +0000214 ... AND Append Memory Consumption To File isTest=True action=test_start
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000215 Run Keyword If ${has_dataplane} Clean Up Linux
TorstenThieme0fa54432023-02-14 13:27:20 +0000216 ${output_file_olt}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptionsOpenOltAdapterOltTest.txt
217 Create File ${output_file_olt} This file contains the memory consumptions of openolt adapter.
218 Append To File ${output_file_olt} \r\nTest: ${TEST NAME}
219 ${output_file_onu}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptionsOpenOnuAdapterOltTest.txt
220 Create File ${output_file_onu} This file contains the memory consumptions of openonu adapter.
221 Append To File ${output_file_onu} \r\nTest: ${TEST NAME}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000222 Run Keyword If ${print2console} Log \r\nStart ${iterations} iterations. console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000223 ${start_mem_consumption_olt}= Get And Write Memory Consumption Per Container To File adapter-open-olt ${NAMESPACE}
224 ... ${output_file_olt} Start settling_memory=True
225 ${start_mem_consumption_onu}= Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE}
226 ... ${output_file_onu} Start
227 ${Device_Setup}= Set Variable True
228 Set Global Variable ${Device_Setup}
229 Setup
230 Power On ONU Device ${NAMESPACE}
231 ${setup_mem_consumption_olt}= Get And Write Memory Consumption Per Container To File adapter-open-olt ${NAMESPACE}
232 ... ${output_file_olt} Setup settling_memory=True
233 Append Memory Consumption To File isTest=True action=test_setup
234 Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE} ${output_file_onu} Setup
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000235 FOR ${I} IN RANGE 1 ${iterations} + 1
236 Run Keyword If ${print2console} Log \r\nStart iteration ${I} of ${iterations}. console=yes
237 Run Keyword If "${workflow}"=="DT" Perform Sanity Test DT
238 ... ELSE IF "${workflow}"=="TT" Perform Sanity Tests TT
239 ... ELSE Perform Sanity Test
240 Sleep 5s
241 Run Keyword If ${print2console} Log Delete OLTs. console=yes
242 Delete Devices In Voltha Type=openolt
243 Run Keyword If ${print2console} Log Check OLTs removed. console=yes
244 Wait Until Keyword Succeeds ${timeout} 1s Test Empty Device List
245 Sleep 20s
246 Run Keyword If ${print2console} Log Add OLTs (calling Setup). console=yes
247 Setup
TorstenThieme0fa54432023-02-14 13:27:20 +0000248 Power On ONU Device ${NAMESPACE}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000249 Run Keyword If ${print2console} Log Wait for ONUs come back. console=yes
250 ${list_onus} Create List
251 Build ONU SN List ${list_onus}
252 Wait Until Keyword Succeeds ${timeout} 1s Check all ONU OperStatus ${list_onus} ACTIVE
253 Build ONU SN List ${list_onus}
254 ${onu_reason}= Set Variable If "${workflow}"=="DT" initial-mib-downloaded
255 ... "${workflow}"=="TT" initial-mib-downloaded
256 ... "${workflow}"=="ATT" omci-flows-pushed
257 Wait Until Keyword Succeeds ${timeout} 1s
258 ... Validate ONU Devices ENABLED ACTIVE REACHABLE ${list_onus} onu_reason=${onu_reason}
TorstenThieme0fa54432023-02-14 13:27:20 +0000259 ${formatedIt}= Format String {:>3} ${I}
260 Get And Write Memory Consumption Per Container To File adapter-open-olt ${NAMESPACE} ${output_file_olt}
261 ... Iteration ${formatedIt}
262 Get And Write Memory Consumption Per Container To File adapter-open-onu ${NAMESPACE} ${output_file_onu}
263 ... Iteration ${formatedIt}
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000264 END
TorstenThieme30c64752023-03-01 10:24:58 +0000265 Wait Until Keyword Succeeds 500s 30s
TorstenThieme0fa54432023-02-14 13:27:20 +0000266 ... Validate Memory Consumptions adapter-open-olt ${NAMESPACE} ${setup_mem_consumption_olt} ${output_file_onu}
267 ... out_string=Validate
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000268 [Teardown] Run Keywords Printout ONU Serial Number and Device Id print2console=${print2console}
269 ... AND Run Keyword If ${logging} Get Logical Id of OLT
270 ... AND Run Keyword If ${logging} Collect Logs
271 ... AND Teardown Test
TorstenThieme0fa54432023-02-14 13:27:20 +0000272 ... AND Append Memory Consumption To File isTest=True compare_to=test_setup
TorstenThieme30c64752023-03-01 10:24:58 +0000273 ... AND Run Keyword And Ignore Error Wait Until Keyword Succeeds 300s 30s
TorstenThieme0fa54432023-02-14 13:27:20 +0000274 ... Validate Memory Consumptions adapter-open-olt ${NAMESPACE} ${start_mem_consumption_olt} ${output_file_olt}
275 ... AND Validate Memory Consumptions adapter-open-onu ${NAMESPACE} ${start_mem_consumption_onu} ${output_file_onu}
276 ... AND Append Memory Consumption To File isTest=True compare_to=test_start
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000277 ... AND Stop Logging MemoryLeakTestOlt
278
279
TorstenThieme700ccbf2022-02-11 10:24:05 +0000280*** Keywords ***
281Setup Suite
282 [Documentation] Set up the test suite
283 Start Logging Setup or Teardown Setup-${SUITE NAME}
284 ${LogInfo}= Catenate
285 ... \r\nPassed arguments:
286 ... debugmode:${debugmode}, logging:${logging}, pausebeforecleanup:${pausebeforecleanup},
287 ... print2console:${print2console}, workflow:${workflow}, kvstoreprefix:${kvstoreprefix},
288 ... iterations:${iterations}
289 Log ${LogInfo} console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000290 Create Global Memory Consumption File
291 ${Device_Setup}= Set Variable False
292 Set Global Variable ${Device_Setup}
293 ${Start_Time}= Get Time epoch
294 Set Global Variable ${Start_Time}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000295 Common Test Suite Setup
296 # set tech profiles
297 ${preload_tech_profile}= Set Variable If ${unitag_sub} and "${workflow}"=="TT" and not ${has_dataplane} True False
298 Set Suite Variable ${preload_tech_profile}
299 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-HSIA ${INFRA_NAMESPACE} 64
300 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-VoIP ${INFRA_NAMESPACE} 65
301 Run Keyword If ${preload_tech_profile} Set Tech Profile TT-multi-uni-MCAST-AdditionalBW-None ${INFRA_NAMESPACE} 66
302 # delete etcd MIB Template Data
303 Delete MIB Template Data ${INFRA_NAMESPACE}
304 # delete etcd onu data
305 Delete ONU Go Adapter ETCD Data namespace=${INFRA_NAMESPACE} validate=True
306 Run Keyword If ${logging} Collect Logs
307 Stop Logging Setup or Teardown Setup-${SUITE NAME}
308
TorstenThieme700ccbf2022-02-11 10:24:05 +0000309Teardown Suite
310 [Documentation] Replaces the Suite Teardown in utils.robot.
311 ... Cleans up and checks all ONU ports disabled in ONOS.
312 ... Furthermore gives the possibility to pause the execution.
313 Start Logging Setup or Teardown Teardown-${SUITE NAME}
TorstenThieme0fa54432023-02-14 13:27:20 +0000314 Append Memory Consumption To File compare_to=suite_setup
315 ${End_Time}= Get Time epoch
316 FOR ${container} IN @{list_of_container}
317 ${mem_consumption}= utility.get_memory_consumptions_range ${prometheusaddr}:${prometheusport} ${container}
318 ... ${namespace} ${Start_Time} ${End_Time}
319 Write Memory Consumption File Per Container ${container} ${mem_consumption}
320 END
TorstenThieme700ccbf2022-02-11 10:24:05 +0000321 Run Keyword If ${pausebeforecleanup} Import Library Dialogs
322 Run Keyword If ${pausebeforecleanup} Pause Execution Press OK to continue with clean up!
TorstenThieme0fa54432023-02-14 13:27:20 +0000323 Run Keyword If ${pausebeforecleanup} Log ${consumption_max} Teardown will be continued... console=yes
324 Run Keyword If ${teardown_device} and ${Device_Setup} Delete All Devices and Verify
TorstenThieme700ccbf2022-02-11 10:24:05 +0000325 Run Keyword Unless ${etcdcheckintestteardown} Wait Until Keyword Succeeds ${timeout} 1s
326 ... Validate Onu Data In Etcd ${INFRA_NAMESPACE} 0 ${kvstoreprefix} without_pm_data=False
TorstenThieme0fa54432023-02-14 13:27:20 +0000327 Run Keyword If ${Device_Setup} Wait for Ports in ONOS for all OLTs ${ONOS_SSH_IP} ${ONOS_SSH_PORT} 0 BBSM ${timeout}
328 Run Keyword If ${logging} Collect Logs
TorstenThieme700ccbf2022-02-11 10:24:05 +0000329 Stop Logging Setup or Teardown Teardown-${SUITE NAME}
330 Close All ONOS SSH Connections
331 Set Suite Variable ${TechProfile} ${EMPTY}
332 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 64
333 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 65
334 Run Keyword If ${preload_tech_profile} Remove Tech Profile ${INFRA_NAMESPACE} 66
335
336Teardown Test
337 [Documentation] Post-test Teardown
TorstenThieme22491b72022-03-31 13:12:27 +0000338 # log ONOS flows after remove check
339 ${flow}= Run Keyword If "${TEST STATUS}"=="FAIL" Execute ONOS CLI Command use single connection
340 ... ${ONOS_SSH_IP} ${ONOS_SSH_PORT} flows -s any ${of_id}
341 Run Keyword If "${TEST STATUS}"=="FAIL" Log ${flow}
TorstenThieme700ccbf2022-02-11 10:24:05 +0000342 Run Keyword If ${pausebeforecleanup} Import Library Dialogs
343 Run Keyword If ${pausebeforecleanup} Pause Execution Press OK to continue with clean up!
344 Run Keyword If ${pausebeforecleanup} Log Teardown will be continued... console=yes
TorstenThieme0fa54432023-02-14 13:27:20 +0000345 Run Keyword If ${teardown_device} and ${Device_Setup} Delete All Devices and Verify
TorstenThieme700ccbf2022-02-11 10:24:05 +0000346 # delete etcd MIB Template Data
347 Delete MIB Template Data ${INFRA_NAMESPACE}
348 # check etcd data are empty
349 Run Keyword If ${etcdcheckintestteardown} Wait Until Keyword Succeeds ${timeout} 1s
350 ... Validate Onu Data In Etcd ${INFRA_NAMESPACE} 0 ${kvstoreprefix} without_pm_data=False
351 Sleep 5s
TorstenThieme1208ab52022-04-11 14:24:27 +0000352
TorstenThieme0fa54432023-02-14 13:27:20 +0000353Create Global Memory Consumption File
354 [Documentation] Creates global memory consumption file and read the start values
355 ${list_of_container}= Create List voltha ofagent adapter-open-olt adapter-open-onu
356 Set Global Variable ${list_of_container}
357 ${global_output_file}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptions${workflow}.txt
358 Create File ${global_output_file} This file contains the memory consumptions of all voltha adapter of workflow ${workflow}.
359 Set Global Variable ${global_output_file}
360 ${time}= Get Time
361 Append To File ${global_output_file} \r\n------------------------------------------------------
362 Append To File ${global_output_file} \r\nMemory consumptions Suite Setup at ${time}
363 Append To File ${global_output_file} \r\n------------------------------------------------------
364 &{suite_setup} Create Dictionary
365 FOR ${container} IN @{list_of_container}
366 ${mem_consumption}= Append Memory Consumption Per Container To File ${container}
367 Set To Dictionary ${suite_setup} ${container} ${mem_consumption}
368 END
369 &{memory_consumption_dict} Create Dictionary suite_setup ${suite_setup}
370 Set Global Variable ${memory_consumption_dict}
371
372Append Memory Consumption To File
373 [Documentation] Appends data to global memory consumption file per container
374 [Arguments] ${isTest}=False ${action}=${EMPTY} ${output_file}=${global_output_file} ${compare_to}=${EMPTY}
375 ${time}= Get Time
376 ${TestOrSuite} Set Variable If ${isTest} Test Suite
377 ${SetupOrTeardown} Set Variable If "${action}"!="${EMPTY}" Setup/Start (${action}) Teardown
378 Append To File ${global_output_file} \r\n------------------------------------------------------
379 Append To File ${global_output_file} \r\nMemory consumptions ${TestOrSuite} ${SetupOrTeardown} at ${time}
380 Run Keyword If ${isTest} Append To File ${global_output_file} \r\nTest: ${TEST NAME}
381 Append To File ${global_output_file} \r\n------------------------------------------------------
382 &{test_setup} Create Dictionary
383 FOR ${container} IN @{list_of_container}
384 ${mem_consumption}= Append Memory Consumption Per Container To File ${container} output_file=${output_file}
385 ... compare_to=${compare_to}
386 Run Keyword If "${action}"!="${EMPTY}" Set To Dictionary ${test_setup} ${container} ${mem_consumption}
387 END
388 Run Keyword If "${action}"!="${EMPTY}" Set To Dictionary ${memory_consumption_dict} ${action} ${test_setup}
389 Run Keyword If "${action}"!="${EMPTY}" Set Global Variable ${memory_consumption_dict}
390
391Append Memory Consumption Per Container To File
392 [Documentation] Appends data to global memory consumption file per container
393 [Arguments] ${container} ${namespace}=${NAMESPACE} ${output_file}=${global_output_file} ${compare_to}=${EMPTY}
TorstenThiemef06257c2023-03-06 12:40:09 +0000394 ${mem_consumption}= Wait Until Keyword Succeeds 300s 5s
TorstenThieme0fa54432023-02-14 13:27:20 +0000395 ... Get Memory Consumptions ${prometheusaddr} ${prometheusport} ${container} ${namespace}
396 ${formated_mem}= Format String {:>10} ${mem_consumption}
397 ${prestring}= Catenate \r\nMemory consumptions of ${container}
398 ${formated_prestring}= Format String {:<43} ${prestring}
399 ${poststring}= Run Keyword If "${compare_to}"!="${EMPTY}" Compare Memory Consumptions ${mem_consumption}
400 ... ${container} ${compare_to}
401 ... ELSE Set Variable ${EMPTY}
402 ${out_string}= Catenate ${formated_prestring} : ${formated_mem} Bytes ${poststring}
403 Append To File ${output_file} ${out_string}
404 [return] ${mem_consumption}
405
406Compare Memory Consumptions
407 [Documentation] Compares the current memory consumptions with the compare-to value of passed container and
408 ... creates corresponding string.
409 [Arguments] ${mem_consumption} ${container} ${compare_to}
410 ${compare_value}= Get From Dictionary ${memory_consumption_dict['${compare_to}']} ${container}
411 ${diff_value}= Evaluate ${mem_consumption}-${compare_value}
412 ${percentage_value}= Evaluate 100*${mem_consumption}/${compare_value}
413 ${percentage_value}= Convert To Number ${percentage_value} 2
414 ${formated_start}= Format String {:>10} ${compare_value}
415 ${formated_diff}= Format String {:>10} ${diff_value}
416 ${formated_perc}= Format String {:>7} ${percentage_value}
417 ${out_string}= Catenate : Corresponds ${formated_perc}% compared to ${compare_to} (${formated_start} Bytes) :
418 ... Difference: ${formated_diff} Bytes
419 [return] ${out_string}
420
421Get And Write Memory Consumption Per Container To File
422 [Documentation] Gets and write current memory consumptions to memory consumption file per container
423 [Arguments] ${container} ${namespace} ${output_file} ${addstring}=${EMPTY} ${settling_memory}=False
424 ${mem_consumption}= Run Keyword If ${settling_memory} Settling Memory Consumptions ${prometheusaddr}
425 ... ${prometheusport} ${container} ${namespace}
426 ... ELSE Wait Until Keyword Succeeds 60s 5s Get Memory Consumptions ${prometheusaddr} ${prometheusport}
427 ... ${container} ${namespace}
428 ${time}= Get Time
429 ${formated_mem}= Format String {:>10} ${mem_consumption}
430 ${prestring}= Catenate Memory consumptions of ${container} ${addstring}
431 ${formated_prestring}= Format String {:<54} ${prestring}
432 ${out_string}= Catenate \r\n${formated_prestring} : ${formated_mem} Bytes at ${time}
433 Append To File ${output_file} ${out_string}
434 Run Keyword If ${print2console} Log ${formated_prestring} : ${formated_mem} Bytes at ${time} console=yes
435 [return] ${mem_consumption}
436
TorstenThieme1208ab52022-04-11 14:24:27 +0000437Check for new ONU Device IDs
438 [Documentation] Checks that no old onu device ids stay
439 [Arguments] ${old_device_ids}
440 ${new_device_ids}= Get ONUs Device IDs from Voltha
441 Should Not Be Empty ${new_device_ids} No new ONU device IDs
442 FOR ${item} IN @{old_device_ids}
443 List Should Not Contain Value ${new_device_ids} ${item} Old device id ${item} still present.
444 END
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000445
446Validate Memory Consumptions
TorstenThieme0fa54432023-02-14 13:27:20 +0000447 [Documentation] Validates memory consumptions of passed POD
448 [Arguments] ${container} ${namespace} ${start_value} ${output_file} ${out_string}=Teardown
449 ${mem_consumption}= Get And Write Memory Consumption Per Container To File ${container} ${namespace}
450 ... ${output_file} ${out_string}
451 ${mem_consumption}= Convert To Number ${mem_consumption} 1
TorstenThieme30c64752023-03-01 10:24:58 +0000452 ${upper_bound}= Evaluate (${start_value} + (${start_value}*0.175))
TorstenThieme7e8ccb92022-09-29 09:12:38 +0000453 Should Be True ${upper_bound} >= ${mem_consumption}
TorstenThieme0fa54432023-02-14 13:27:20 +0000454
455Settling Memory Consumptions
456 [Documentation] Delivers memory consumptions of passed POD after memory consumptions are leveled.
457 ... - collecting memory consumption at least about 5 minutes, but max 20 minutes
458 ... - built average of memory consumtions
TorstenThieme30c64752023-03-01 10:24:58 +0000459 ... - wait current value does not deviate from the average by more than 12%.
TorstenThieme0fa54432023-02-14 13:27:20 +0000460 ... - deliver average value value
461 [Arguments] ${prometheusaddr} ${prometheusport} ${container} ${namespace}
462 @{consumption_list}= Create List
463 ${consumption_sum}= Set Variable 0
464 ${consumption_max}= Set Variable 0
465 ${average_value}= Set Variable 0
466 FOR ${index} IN RANGE 1 21
467 ${current_consumptions}= Get Memory Consumptions ${prometheusaddr} ${prometheusport} ${container} ${namespace}
468 Append To List ${consumption_list} ${current_consumptions}
469 ${consumption_sum}= Evaluate ${consumption_sum}+${current_consumptions}
470 ${consumption_max}= Set Variable If ${current_consumptions}>${consumption_max} ${current_consumptions}
471 ... ${consumption_max}
472 ${average_value}= Evaluate ${consumption_sum}/${index}
473 ${upper_bound}= Evaluate (${average_value} + (${average_value}*0.12))
474 ${lower_bound}= Evaluate (${average_value} - (${average_value}*0.12))
475 ${time}= Get Time
476 ${formatedIt}= Format String {:>3} ${index}
477 ${formated_mem}= Format String {:>10} ${current_consumptions}
478 ${prestring}= Catenate Memory consumptions of ${container} Leveling ${formatedIt}
479 ${formated_prestring}= Format String {:<54} ${prestring}
480 Run Keyword If ${print2console} Log ${formated_prestring} : ${formated_mem} Bytes at ${time} console=yes
481 Exit For Loop If ${index}>5 and ${current_consumptions}<${upper_bound} and ${current_consumptions}>${lower_bound}
482 Sleep 60s
483 END
484 [return] ${average_value}
485
486Get Memory Consumptions
487 [Documentation] Delivers memory consumptions of passed POD
488 [Arguments] ${prometheusaddr} ${prometheusport} ${container} ${namespace}
489 ${mem_consumption}= utility.get_memory_consumptions ${prometheusaddr}:${prometheusport} ${container} ${namespace}
TorstenThieme30c64752023-03-01 10:24:58 +0000490 Should Be True ${mem_consumption} > 0
TorstenThieme0fa54432023-02-14 13:27:20 +0000491 [return] ${mem_consumption}
492
493Write Memory Consumption File Per Container
494 [Documentation] Writes memory consumptions file of passed POD for later evaluation.
495 [Arguments] ${container} ${mem_consumption}
496 ${output_file}= Catenate SEPARATOR=/ ${OUTPUT DIR} MemoryConsumptions${workflow}${container}.txt
497 Create File ${output_file}
498 FOR ${value} IN @{mem_consumption}
499 ${epoch}= Convert To String ${value[0]}
500 Append To File ${output_file} ${epoch},${value[1]}${\n}
501 END