blob: b1f0f82df2cfed5c8e09b82665d00c3728b45800 [file] [log] [blame]
ajayf66b38e2021-12-09 23:44:54 -08001..
2 SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org>
3 SPDX-License-Identifier: Apache-2.0
4
ajayce0380c2021-12-14 21:51:00 -08005.. _sdcore-testing:
6
7Test frameworks and coverage
8============================
ajayf66b38e2021-12-09 23:44:54 -08009
10Test Framework
11--------------
12
13NG40
14""""
15
16NG40 tool is used as RAN emulator in SD-Core testing. NG40 runs inside a VM
17which is connected to both Aether control plane and data plane. In testing
18scenarios that involve data plane verification, NG40 also emulates a few
19application servers which serve as the destinations of data packets.
20
21A typical NG40 test case involves UE attaching, data plane verification and
22UE detaching. During the test NG40 acts as UEs and eNBs and talks to the
23mobile core to complete attach procedures for each UE it emulates. Then NG40
24verifies that data plane works for each attached UE by sending traffic between
25UEs and application servers. Before finishing each test NG40 performs detach
26procedures for each attached UE.
27
28Test cases
29''''''''''
30
31Currently the following NG40 test cases are implemented:
32
334G Tests:
34
351. ``4G_M2AS_PING_FIX`` (attach, dl ping, detach)
362. ``4G_M2AS_UDP`` (attach, dl+ul udp traffic, detach)
373. ``4G_M2AS_TCP`` (attach, release, service request, dl+ul tcp traffic, detach)
384. ``4G_AS2M_PAGING`` (attach, release, dl udp traffic, detach)
395. ``4G_M2AS_SRQ_UDP`` (attach, release, service request, dl+ul udp traffic)
406. ``4G_M2CN_PS`` (combined IMSI/PTMSI attach, detach)
417. ``4G_HO`` (attach, relocate and dl ping, detach)
428. ``4G_SCALE`` (attach, dl ping, detach with multiple UEs)
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530439. ``4G_3_APPS`` (4G attach-detach with ICMP/UDP Data test with 3 applications)
4410. ``4G_SCALE_HO`` (attach, Handover, dl ping, detach with multiple UEs)
4511. ``4G_ATTACH_STRESS`` (attach-detach 200 times consecutively)
4612. ``4G_TAU`` (attach, Tracking Area Update, detach)
ajayf66b38e2021-12-09 23:44:54 -080047
485G Tests:
49
501. ``5G_SA_Register_Deregister`` (registration, deregistration)
512. ``5G_SA_Register`` (registration, session establishment, deregistration)
523. ``5G_SA_Release`` (registration, session establishment, dl ping, release, deregistration)
534. ``5G_SA_Activate_Release`` (registration, session establishment, dl ping, release, service request,
54 dl ping, deregistration)
555. ``5G_SA_Scale`` (registration, session establishment, dl ping, deregistration for multiple UEs)
566. ``5G_SA_M2AS_ICMP`` (registration, session establishment, dl ping, deregistration)
577. ``5G_SA_M2AS_TCP`` (registration, session establishment, dl+ul tcp traffic, deregistration)
588. ``5G_SA_M2AS_UDP`` (registration, session establishment, dl+ul udp traffic, deregistration)
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530599. ``5G_SA_SCALE`` (registration, session establishment, dl+ul udp traffic, deregistration with multiple UEs)
6010. ``5G_SA_Error_Malformed_Reg_Type`` (Error Scenario: registration with malformed registration type)
6111. ``5G_SA_Error_Auth_response_with_invalid_RES`` (Error Scenario: Invalid RES sent in Auth Response)
ajayf66b38e2021-12-09 23:44:54 -080062
63All the test cases are parameterized and can take arguments to specify number
64of UEs, attach/detach rate, traffic type/rate etc. For example, ``4G_SCALE``
65test case can be configured as a mini scalability test which performs only 5
66UE attaches in a patchset pre-merge test, while in the nightly tests it can
67take different arguments to run 10K UE attaches with a high attach rate.
68
69Test suites
70'''''''''''
71
72The test cases are atomic testing units and can be combined to build test
73suites. The following test suites have been built so far:
74
751. ``functionality test suite`` verifies basic functionality of the
76 mobile core. 4G functionality suite runs 4G test case #1 to #8 including
77 ``4G_SCALE`` which attaches 5 UEs with 1/s attach rate. 5G functionality
78 suite runs 5G test case #1 to #8 including ``5G_SA_Scale`` which attaches
79 100 UEs with 1/s attach rate.
802. ``scalability test suite`` tests the system by scale and verifies
81 system stability. It runs ``4G_SCALE`` (or ``5G_SA_Scale``) which attaches
82 a large number of UEs with high attach rate (16k UEs with 100/s rate on 4G
83 CI pod, 1k UEs with 10/s rate on 4G staging pod, and 1k UEs with 1/s rate
84 on 5G CI pod).
853. ``performance test suite`` measures performance of the control and
86 data plane. It runs ``4G_SCALE`` multiple times with different attach rates
87 to understand how the system performs under different loads.
88
89Robot Framework
90"""""""""""""""
91
92Robot Framework was chosen to build test cases that involve interacting with
93not only NG40 but also other parts of the system. In these scenarios Robot
94Framework acts as a high level orchestrator which drives various components
95of the system using component specific libraries including NG40.
96
97Currently the ``Integration test suite`` is implemented using Robot
98Framework. In the integration tests Robot Framework calls the ng40 library to
99perform normal attach/detach procedures. Meanwhile it injects failures into
100the system (container restarts, link down etc.) by calling functions
101implemented in the k8s library.
102
103The following integration tests are implemented at the moment:
104
Ruchit Bawa5606dd22021-12-15 23:41:43 +05301054G Tests:
106
1071. Subscriber Attach with HSS Restart (attach, Restart HSS, detach)
1082. Subscriber Attach with SPGWC Restart (attach, Restart SPGWC, detach)
1093. Subscriber Attach Detach HSS Restart Attach Detach (attach-detach, Restart HSS, attach-detach)
1104. Subscriber Attach Detach SPGWC Restart Attach Detach (attach-detach, Restart SPGWC, attach-detach)
1115. Measure Downtime for HSS Restart (Verifies if HSS restarts within acceptable time limit)
1126. Measure Downtime for SPGWC Restart (Verifies if SPGWC restarts within acceptable time limit)
113
1145G Tests:
115
1161. ``Subscriber Register-Deregister with SMF Restart`` (registration, SMF Restart, deregistration)
1172. ``Subscriber Register-Deregister with AUSF Restart`` (registration, AUSF Restart, deregistration)
1183. ``Subscriber Register-Deregister with NRF Restart`` (registration, NRF Restart, deregistration)
1194. ``Subscriber Register-Deregister with NSSF Restart`` (registration, NSSF Restart, deregistration)
1205. ``Subscriber Register-Deregister with PCF Restart`` (registration, PCF Restart, deregistration)
1216. ``Subscriber Register-Deregister with UDR Restart`` (registration, UDR Restart, deregistration)
1227. ``Measure Downtime for SMF Restart`` (Verifies if SMF restarts within acceptable time limit)
1238. ``Measure Downtime for AUSF Restart`` (Verifies if AUSF restarts within acceptable time limit)
1249. ``Measure Downtime for NRF Restart`` (Verifies if NRF restarts within acceptable time limit)
12510. ``Measure Downtime for NSSF Restart`` (Verifies if NSSF restarts within acceptable time limit)
12611. ``Measure Downtime for PCF Restart`` (Verifies if PCF restarts within acceptable time limit)
12712. ``Measure Downtime for UDR Restart`` (Verifies if UDR restarts within acceptable time limit)
12813. ``Context Replacement at AMF`` (Verifies context replacement at AMF after NG40 restart)
12914. ``Context Replacement at UPF`` (Verifies context replacement at UPF after SMF restart)
ajayf66b38e2021-12-09 23:44:54 -0800130
131.. Note::
132 More integration tests are being developed as part of Robot Framework
133
134Test Schedules
135--------------
136
137Nightly Tests
138"""""""""""""
139
140SD-Core nightly tests are a set of jobs managed by Aether Jenkins.
141All four test suites we mentioned above are scheduled to run nightly.
142
1431. ``functionality job (func)`` runs NG40 test cases included in the
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530144 functionality suite and Integration test suite and verifies all tests pass.
ajayf66b38e2021-12-09 23:44:54 -08001452. ``scalability job (scale)`` runs the scalability test suite and reports
146 the number of successful/failed attaches, detaches and pings.
ajayf66b38e2021-12-09 23:44:54 -0800147
148And all these jobs can be scheduled on any of the Aether PODs including
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530149``ci-4g`` pod, ``ci-5g`` pod, ``qa`` pod and ``qa2`` pod. By combining
ajayf66b38e2021-12-09 23:44:54 -0800150the test type and test pod the following Jenkins jobs are generated:
151
Ruchit Bawa5606dd22021-12-15 23:41:43 +05301521. ``ci-4g`` pod: `sdcore_ci-4g_4g_bess_func`, `sdcore_ci-4g_4g_bess_scale`
1532. ``ci-5g`` pod: `sdcore_ci-5g_5g_bess_func`, `sdcore_ci-5g_5g_bess_scale`
1543. ``qa`` pod: `sdcore_qa_4g_bess_func`, `sdcore_qa_4g_bess_scale`
1554. ``qa2`` pod: `sdcore_qa2_4g_bess_func`, `sdcore_qa2_4g_bess_scale`
ajayf66b38e2021-12-09 23:44:54 -0800156
157Nightly Job structure
158"""""""""""""""""""""
159
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530160Take `sdcore_ci-4g_4g_bess_scale` job as an example. It runs the following downstream jobs:
ajayf66b38e2021-12-09 23:44:54 -0800161
Ruchit Bawa5606dd22021-12-15 23:41:43 +05301621. `sdcore_ci-4g_deploy`: this job re-deploys the ``ci-4g`` pod with latest OMEC images.
ajayf66b38e2021-12-09 23:44:54 -0800163
164.. Note::
165 only the ``ci-4g`` and ``ci-5g`` pod jobs trigger deployment downstream job. No
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530166 re-deployment is performed on the qa and qa2 pods before the tests.
ajayf66b38e2021-12-09 23:44:54 -0800167
Ruchit Bawa5606dd22021-12-15 23:41:43 +05301682. `sdcore_ci-4g_4g_bess_robot-test`: this job executes the scalability test suite.
ajayf66b38e2021-12-09 23:44:54 -08001693. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
1704. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
171 test data to database. It also generates plots using Rscript for func and
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530172 scale tests.
ajayf66b38e2021-12-09 23:44:54 -0800173
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530174The integration tests are written using Robot Framework and are executed along
175 with the functional tests. Take `sdcore_ci-4g_4g_bess_func` as an example. It runs the
176 following downstream jobs:
ajayf66b38e2021-12-09 23:44:54 -0800177
Ruchit Bawa5606dd22021-12-15 23:41:43 +05301781. `sdcore_ci-4g_deploy`: this job executes the scalability test suite.
1792. `sdcore_ci-4g_4g_bess_robot-test`: this job calls robot
ajayf66b38e2021-12-09 23:44:54 -0800180 framework to execute the test cases and publishes the test results using
181 `RobotPublisher` Jenkins plugin. The robot results will also be copied to
182 the upstream job and published there.
1833. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
1844. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
185 test data to database. It also generates plots using Rscript for func and
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530186 scale tests.
ajayf66b38e2021-12-09 23:44:54 -0800187
188Patchset Tests
189--------------
190
191SD-Core pre-merge verification covers the following public Github repos: ``c3po``,
192``Nucleus``, ``upf-epc`` and the following private Github repos: ``spgw``. ``amf``,
193``smf``, ``ausf``, ``nssf``, ``nrf``, ``pcf``, ``udm``, ``udr``, ``webconsole``.
194SD-Core CI verifies the following:
195
1961. ONF CLA verification
1972. License verification (FOSSA/Reuse)
1983. NG40 tests
199
200These jobs are automatically triggered by submitted or updated PR to the repos
201above. They can also be triggered manually by commenting ``retest this please``
202to the PR. At this moment only CLI and NG40 verification are mandatory.
203
204The NG40 verification are a set of jobs running on both opencord Jenkins and
205Aether Jenkins (private). The jobs run on opencord Jenkins include
206
2071. `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ (public)
2082. `omec_Nucleus_container_remote <https://jenkins.opencord.org/job/omec_Nucleus_container_remote/>`_ (public)
2093. `omec_upf-epc_container_remote <https://jenkins.opencord.org/job/omec_upf-epc_container_remote/>`_ (public)
2104. `omec_spgw_container_remote` (private, under member-only folder)
211
212And the jobs run on Aether Jenkins include
213
Ruchit Bawa5606dd22021-12-15 23:41:43 +05302141. `c3po_premerge_ci-4g_4g_bess`
2152. `Nucleus_premerge_ci-4g_4g_bess`
2163. `upf-epc_premerge_ci-4g_4g_bess`
2174. `spgw_premerge_ci-4g_4g_bess`
2185. `amf_premerge_ci-5g_5g_bess`
2196. `smf_premerge_ci-5g_5g_bess`
2207. `ausf_premerge_ci-5g_5g_bess`
2218. `nssf_premerge_ci-5g_5g_bess`
2229. `nrf_premerge_ci-5g_5g_bess`
22310. `pcf_premerge_ci-5g_5g_bess`
22411. `udm_premerge_ci-5g_5g_bess`
22512. `udr_premerge_ci-5g_5g_bess`
22613. `webconsole_premerge_ci-5g_5g_bess`
ajayf66b38e2021-12-09 23:44:54 -0800227
228Patchset Job structure
229""""""""""""""""""""""
230
231Take ``c3po`` jobs as an example. ``c3po`` PR triggers a public job
232`omec_c3po_container_remote
233<https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ job running
234on opencord Jenkins through Github webhooks, which then triggers a private job
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530235`c3po_premerge_ci-4g_4g_bess` running on Aether Jenkins using a Jenkins plugin called
ajayf66b38e2021-12-09 23:44:54 -0800236`Parameterized Remote Trigger Plugin
237<https://www.jenkins.io/doc/pipeline/steps/Parameterized-Remote-Trigger/>`_.
238
239The private ``c3po`` job runs the following downstream jobs sequentially:
240
2411. `docker-publish-github_c3po`: this job downloads the ``c3po`` PR, runs docker
242 build and publishes the ``c3po`` docker images to `Aether registry`.
Ruchit Bawa5606dd22021-12-15 23:41:43 +05302432. `sdcore_ci-4g_deploy`: this job deploys the images built from previous job onto
ajayf66b38e2021-12-09 23:44:54 -0800244 the omec ``ci-4g`` pod.
Ruchit Bawa5606dd22021-12-15 23:41:43 +05302453. `sdcore_ci-4g_4g_bess_robot-test`: this job executes the functionality test suite.
ajayf66b38e2021-12-09 23:44:54 -08002464. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
247
Ruchit Bawa5606dd22021-12-15 23:41:43 +0530248After all the downstream jobs are finished, the upstream job (`c3po_premerge_ci-4g_4g_bess`)
ajayf66b38e2021-12-09 23:44:54 -0800249copies artifacts including k8s/container/ng40 logs and pcap files from
250downstream jobs and saves them as Jenkins job artifacts.
251
252These artifacts are also copied to and published by the public job
253(`omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_)
254on opencord Jenkins so that they can be accessed by the OMEC community.
255
256Pre-merge jobs for other SD-Core repos share the same structure.
257
258Post-merge
259""""""""""
260
261The following jobs are triggered as post-merge jobs when PRs are merged to
262SD-Core repos:
263
2641. `docker-publish-github-merge_c3po`
2652. `docker-publish-github-merge_Nucleus`
2663. `docker-publish-github-merge_upf-epc`
2674. `docker-publish-github-merge_spgw`
2685. `docker-publish-github-merge_amf`
2696. `docker-publish-github-merge_smf`
2707. `docker-publish-github-merge_ausf`
2718. `docker-publish-github-merge_nssf`
2729. `docker-publish-github-merge_nrf`
27310. `docker-publish-github-merge_pcf`
27411. `docker-publish-github-merge_udm`
27512. `docker-publish-github-merge_udr`
27613. `docker-publish-github-merge_webconsole`
277
278Again take the ``c3po`` job as an example. The post-merge job (`docker-publish-github-merge_c3po`)
279runs the following downstream jobs sequentially:
280
2811. `docker-publish-github_c3po`: this is the same job as the one in pre-merge
282 section. It checks out the latest ``c3po`` code, runs docker build and
283 publishes the ``c3po`` docker images to `docker hub <https://hub.docker.com/u/omecproject>`__.
284
285.. Note::
286 the images for private repos are published to Aether registry instead of docker hub
287
2882. `c3po_postrelease`: this job submits a patchset to aether-pod-configs repo
289 for updating the CD pipeline with images published in the job above.
290
291Post-merge jobs for other SD-Core repos share the same structure.