Update sd-core testing doc
Change-Id: Idbb0620f499e8f9719e05ef3a265fdde0510c11b
diff --git a/testing/sdcore_testing.rst b/testing/sdcore_testing.rst
index 22b5f3a..73613c4 100644
--- a/testing/sdcore_testing.rst
+++ b/testing/sdcore_testing.rst
@@ -31,14 +31,28 @@
Currently the following NG40 test cases are implemented:
+4G Tests:
+
1. ``4G_M2AS_PING_FIX`` (attach, dl ping, detach)
2. ``4G_M2AS_UDP`` (attach, dl+ul udp traffic, detach)
3. ``4G_M2AS_TCP`` (attach, relaese, service request, dl+ul tcp traffic, detach)
4. ``4G_AS2M_PAGING`` (attach, release, dl udp traffic, detach)
5. ``4G_M2AS_SRQ_UDP`` (attach, release, service request, dl+ul udp traffic)
6. ``4G_M2CN_PS`` (combined IMSI/PTMSI attach, detach)
-7. ``4G_HO`` (attach, relocate and ping, detach)
-8. ``4G_SCALE`` (attach with multiple UEs, ping, detach)
+7. ``4G_HO`` (attach, relocate and dl ping, detach)
+8. ``4G_SCALE`` (attach, dl ping, detach with multiple UEs)
+
+5G Tests:
+
+1. ``5G_SA_Register_Deregister`` (registration, deregistration)
+2. ``5G_SA_Register`` (registration, session establishment, deregistration)
+3. ``5G_SA_Release`` (registration, session establishment, dl ping, release, deregistration)
+4. ``5G_SA_Activate_Release`` (registration, session establishment, dl ping, release, service request,
+ dl ping, deregistration)
+5. ``5G_SA_Scale`` (registration, session establishment, dl ping, deregistration for multiple UEs)
+6. ``5G_SA_M2AS_ICMP`` (registration, session establishment, dl ping, deregistration)
+7. ``5G_SA_M2AS_TCP`` (registration, session establishment, dl+ul tcp traffic, deregistration)
+8. ``5G_SA_M2AS_UDP`` (registration, session establishment, dl+ul udp traffic, deregistration)
All the test cases are parameterized and can take arguments to specify number
of UEs, attach/detach rate, traffic type/rate etc. For example, ``4G_SCALE``
@@ -53,12 +67,15 @@
suites. The following test suites have been built so far:
1. ``functionality test suite`` verifies basic functionality of the
- mobile core. It runs test case #1 to #8 including ``4G_SCALE`` which attaches
- 5 UEs with 1/s attach rate
+ mobile core. 4G functionality suite runs 4G test case #1 to #8 including
+ ``4G_SCALE`` which attaches 5 UEs with 1/s attach rate. 5G functionality
+ suite runs 5G test case #1 to #8 including ``5G_SA_Scale`` which attaches
+ 100 UEs with 1/s attach rate.
2. ``scalability test suite`` tests the system by scale and verifies
- system stability. It runs ``4G_SCALE`` which attaches a large number of UEs
- with high attach rate (16k UEs with 100/s rate on dev pod, and 10k UEs with
- 10/s rate on staging pod)
+ system stability. It runs ``4G_SCALE`` (or ``5G_SA_Scale``) which attaches
+ a large number of UEs with high attach rate (16k UEs with 100/s rate on 4G
+ CI pod, 1k UEs with 10/s rate on 4G staging pod, and 1k UEs with 1/s rate
+ on 5G CI pod).
3. ``performance test suite`` measures performance of the control and
data plane. It runs ``4G_SCALE`` multiple times with different attach rates
to understand how the system performs under different loads.
@@ -107,42 +124,43 @@
SCTP heartbeat RTT, GTP ICMP RTT and call setup latency numbers.
And all these jobs can be scheduled on any of the Aether PODs including
-``dev`` pod, ``staging`` pod and ``qa`` pod. By combining the test type and
-test pod the following Jenkins jobs are generated:
+``ci-4g`` pod, ``ci-5g`` pod, ``staging`` pod and ``qa`` pod. By combining
+the test type and test pod the following Jenkins jobs are generated:
-1. ``dev`` pod: `func_dev`, `scale_dev`, `perf_dev`, `integ_dev`
+1. ``ci-4g`` pod: `sdcore_func_ci-4g`, `sdcore_scale_ci-4g`, `sdcore_perf_ci-4g`, `sdcore_integ_ci-4g`
+1. ``ci-5g`` pod: `sdcore_func_ci-5g`, `sdcore_scale_ci-5g`
2. ``staging`` pod: `func_staging`, `scale_staging`, `perf_staging`, `integ_staging`
3. ``qa`` pod: `func_qa`, `scale_qa`, `perf_qa`, `integ_qa`
Job structure
^^^^^^^^^^^^^
-Take `scale_dev` job as an example. It runs the following downstream jobs:
+Take `sdcore_scale_ci-4g` job as an example. It runs the following downstream jobs:
-1. `omec_deploy_dev`: this job re-deploys the dev pod with latest OMEC images.
+1. `omec_deploy_ci-4g`: this job re-deploys the ci-4g pod with latest OMEC images.
.. Note::
- only the dev pod job triggers a deployment downstream job. No
+ only the ci-4g and ci-5g pod jobs trigger deployment downstream job. No
re-deployment is performed on the staging and qa pod before the tests
-2. `ng40-test_dev`: this job executes the scalability test suite.
-3. `archive-artifacts_dev`: this job collects and uploads k8s and container logs.
-4. `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the
+2. `ng40-test_ci-4g`: this job executes the scalability test suite.
+3. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
+4. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
test data to database. It also generates plots using Rscript for func and
scale tests
The integration tests are written using Robot Framework so have a slightly
-different Jenkins Job structure. Take `integ_dev` as an example. It runs the
+different Jenkins Job structure. Take `sdcore_integ_ci-4g` as an example. It runs the
following downstream jobs:
-1. `omec_deploy_dev`: this job executes the scalability test suite.
-2. `robotframework-test_dev`: this job is similar to `ng40-test_dev` with the
+1. `omec_deploy_ci-4g`: this job executes the scalability test suite.
+2. `robotframework-test_ci-4g`: this job is similar to `ng40-test_ci-4g` with the
exception that instead of directly executing NG40 commands it calls robot
framework to execute the test cases and publishes the test results using
`RobotPublisher` Jenkins plugin. The robot results will also be copied to
the upstream job and published there.
-3. `archive-artifacts_dev`: this job collects and uploads k8s and container logs.
-4. `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the
+3. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
+4. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
test data to database. It also generates plots using Rscript for func and
scale tests
@@ -152,9 +170,10 @@
Overview
^^^^^^^^
-SD-Core pre-merge verifications cover the following Github repos: ``c3po``,
-``Nucleus``, ``upf-epc`` and ``spgw`` (private). OMEC CI includes the following
-verifications:
+SD-Core pre-merge verifications cover the following public Github repos: ``c3po``,
+``Nucleus``, ``upf-epc`` and the following private Github repos: ``spgw``. ``amf``,
+``smf``, ``ausf``, ``nssf``, ``nrf``, ``pcf``, ``udm``, ``udr``, ``webconsole``.
+SD-Core CI includes the following verifications:
1. ONF CLA verification
2. License verifications (FOSSA/Reuse)
@@ -175,29 +194,38 @@
And the jobs run on Aether Jenkins include
-1. `c3po_premerge_dev`
-2. `Nucleus_premerge_dev`
-3. `upf-epc_premerge_dev`
-4. `spgw_premerge_dev`
+1. `c3po_premerge_ci-4g`
+2. `Nucleus_premerge_ci-4g`
+3. `upf-epc_premerge_ci-4g`
+4. `spgw_premerge_ci-4g`
+5. `amf_premerge_ci-5g`
+6. `smf_premerge_ci-5g`
+7. `ausf_premerge_ci-5g`
+8. `nssf_premerge_ci-5g`
+9. `nrf_premerge_ci-5g`
+10. `pcf_premerge_ci-5g`
+11. `udm_premerge_ci-5g`
+12. `udr_premerge_ci-5g`
+13. `webconsole_premerge_ci-5g`
Job structure
^^^^^^^^^^^^^
Take c3po jobs as an example. c3po PR triggers a public job `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__
job running on opencord Jenkins through Github webhooks,
-which then triggers a private job `c3po_premerge_dev` running on Aether Jenkins
+which then triggers a private job `c3po_premerge_ci-4g` running on Aether Jenkins
using a Jenkins plugin called `Parameterized Remote Trigger Plugin <https://www.jenkins.io/doc/pipeline/steps/Parameterized-Remote-Trigger/>`__.
The private c3po job runs the following downstream jobs sequentially:
1. `docker-publish-github_c3po`: this job downloads the c3po PR, runs docker
build and publishes the c3po docker images to `Aether registry`.
-2. `omec_deploy_dev`: this job deploys the images built from previous job onto
- the omec dev pod.
-3. `ng40-test_dev`: this job executes the functionality test suite.
-4. `archive-artifacts_dev`: this job collects and uploads k8s and container logs.
+2. `omec_deploy_ci-4g`: this job deploys the images built from previous job onto
+ the omec ci-4g pod.
+3. `ng40-test_ci-4g`: this job executes the functionality test suite.
+4. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
-After all the downstream jobs are finished, the upstream job (`c3po_premerge_dev`)
+After all the downstream jobs are finished, the upstream job (`c3po_premerge_ci-4g`)
copies artifacts including k8s/container/NG40 logs and pcap files from
downstream jobs and saves them as Jenkins job artifacts.
@@ -205,18 +233,27 @@
(`omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__)
on opencord Jenkins so that they can be accessed by the OMEC community.
-Pre-merge jobs for other OMEC repos share the same structure.
+Pre-merge jobs for other SD-Core repos share the same structure.
Post-merge
^^^^^^^^^^
The following jobs are triggered as post-merge jobs when PRs are merged to
-OMEC repos:
+SD-Core repos:
1. `docker-publish-github-merge_c3po`
2. `docker-publish-github-merge_Nucleus`
3. `docker-publish-github-merge_upf-epc`
4. `docker-publish-github-merge_spgw`
+5. `docker-publish-github-merge_amf`
+6. `docker-publish-github-merge_smf`
+7. `docker-publish-github-merge_ausf`
+8. `docker-publish-github-merge_nssf`
+9. `docker-publish-github-merge_nrf`
+10. `docker-publish-github-merge_pcf`
+11. `docker-publish-github-merge_udm`
+12. `docker-publish-github-merge_udr`
+13. `docker-publish-github-merge_webconsole`
Again take the c3po job as an example. The post-merge job (`docker-publish-github-merge_c3po`)
runs the following downstream jobs sequentially:
@@ -226,9 +263,9 @@
publishes the c3po docker images to `docker hub <https://hub.docker.com/u/omecproject>`__.
.. Note::
- the spgw images are published to Aether registry instead of docker hub
+ the images for private repos are published to Aether registry instead of docker hub
2. `c3po_postrelease`: this job submits a patchset to aether-pod-configs repo
for updating the CD pipeline with images published in the job above.
-Post-merge jobs for other OMEC repos share the same structure.
+Post-merge jobs for other SD-Core repos share the same structure.