You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | SD-Core Testing |
| 6 | =============== |
| 7 | |
| 8 | Test Framework |
| 9 | -------------- |
| 10 | |
| 11 | NG40 |
| 12 | ~~~~ |
| 13 | |
| 14 | Overview |
| 15 | ^^^^^^^^ |
| 16 | |
| 17 | NG40 tool is used as RAN emulator in SD-Core testing. NG40 runs inside a VM |
| 18 | which is connected to both Aether control plane and data plane. In testing |
| 19 | scenarios that involve data plane verifications, NG40 also emulates a few |
| 20 | application servers which serve as the destinations of data packets. |
| 21 | |
| 22 | A typical NG40 test case involves UE attaching, data plane verifications and |
| 23 | UE detaching. During the test NG40 acts as UEs and eNBs and talks to the |
| 24 | mobile core to complete attach procedures for each UE it emulates. Then NG40 |
| 25 | verifies that data plane works for each attached UE by sending traffic between |
| 26 | UEs and application servers. Before finishing each test NG40 performs detach |
| 27 | procedures for each attached UE. |
| 28 | |
| 29 | Test cases |
| 30 | ^^^^^^^^^^ |
| 31 | |
| 32 | Currently the following NG40 test cases are implemented: |
| 33 | |
| 34 | 1. ``4G_M2AS_PING_FIX`` (attach, dl ping, detach) |
| 35 | 2. ``4G_M2AS_UDP`` (attach, dl+ul udp traffic, detach) |
| 36 | 3. ``4G_M2AS_TCP`` (attach, relaese, service request, dl+ul tcp traffic, detach) |
| 37 | 4. ``4G_AS2M_PAGING`` (attach, release, dl udp traffic, detach) |
| 38 | 5. ``4G_M2AS_SRQ_UDP`` (attach, release, service request, dl+ul udp traffic) |
| 39 | 6. ``4G_M2CN_PS`` (combined IMSI/PTMSI attach, detach) |
| 40 | 7. ``4G_HO`` (attach, relocate and ping, detach) |
| 41 | 8. ``4G_SCALE`` (attach with multiple UEs, ping, detach) |
| 42 | |
| 43 | All the test cases are parameterized and can take arguments to specify number |
| 44 | of UEs, attach/detach rate, traffic type/rate etc. For example, ``4G_SCALE`` |
| 45 | test case can be configured as a mini scalability test which performs only 5 |
| 46 | UE attaches in a patchset pre-merge test, while in the nightly tests it can |
| 47 | take different arguments to run 10K UE attaches with a high attach rate. |
| 48 | |
| 49 | Test suites |
| 50 | ^^^^^^^^^^^ |
| 51 | |
| 52 | The test cases are atomic testing units and can be combined to build test |
| 53 | suites. The following test suites have been built so far: |
| 54 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 55 | 1. ``functionality test suite`` verifies basic functionality of the |
| 56 | mobile core. It runs test case #1 to #8 including ``4G_SCALE`` which attaches |
| 57 | 5 UEs with 1/s attach rate |
| 58 | 2. ``scalability test suite`` tests the system by scale and verifies |
| 59 | system stability. It runs ``4G_SCALE`` which attaches a large number of UEs |
| 60 | with high attach rate (16k UEs with 100/s rate on dev pod, and 10k UEs with |
| 61 | 10/s rate on staging pod) |
| 62 | 3. ``performance test suite`` measures performance of the control and |
| 63 | data plane. It runs ``4G_SCALE`` multiple times with different attach rates |
| 64 | to understand how the system performs under different loads. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 65 | |
| 66 | Robot Framework |
| 67 | ~~~~~~~~~~~~~~~ |
| 68 | |
| 69 | Robot Framework was chosen to build test cases that involve interacting with |
| 70 | not only NG40 but also other parts of the system. In these scenarios Robot |
| 71 | Framework acts as a high level orchestrator which drives various components |
| 72 | of the system using component specific libraries including NG40. |
| 73 | |
| 74 | Currently the ``Integration test suite`` is implemented using Robot |
| 75 | Framework. In the integration tests Robot Framework calls ng40 library to |
| 76 | perform normal attach/detach procedures. Meanwhile it injects failures into |
| 77 | the system (container restarts, link down etc.) by calling functions |
| 78 | implemented in the k8s library. |
| 79 | |
| 80 | The following integration tests are implemented at the moment: |
| 81 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 82 | 1. Subscriber Attach with HSS Restart |
| 83 | 2. Subscriber Attach with MME Restart |
| 84 | 3. Subscriber Attach with SPGWC Restart |
| 85 | 4. Subscriber Attach with PFCP Agent Restart |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 86 | |
| 87 | .. Note:: |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 88 | More integration tests are being developed as part of Robot Framework |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 89 | |
| 90 | Test Schedules |
| 91 | -------------- |
| 92 | |
| 93 | Nightly Tests |
| 94 | ~~~~~~~~~~~~~ |
| 95 | |
| 96 | Overview |
| 97 | ^^^^^^^^ |
| 98 | |
| 99 | SD-Core nightly tests are a set of jobs managed by Aether Jenkins. |
| 100 | All four test suites we mentioned above are scheduled to run nightly. |
| 101 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 102 | 1. ``functionality job (func)`` runs NG40 test cases included in the |
| 103 | functionality suite and verifies all tests pass. |
| 104 | 2. ``scalability job (scale)`` runs the scalability test suite and reports |
| 105 | the number of successful/failed attaches, detaches and pings. |
| 106 | 3. ``performance job (perf)`` runs the performance test suite and reports |
| 107 | SCTP heartbeat RTT, GTP ICMP RTT and call setup latency numbers. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 108 | |
| 109 | And all these jobs can be scheduled on any of the Aether PODs including |
| 110 | ``dev`` pod, ``staging`` pod and ``qa`` pod. By combining the test type and |
| 111 | test pod the following Jenkins jobs are generated: |
| 112 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 113 | 1. ``dev`` pod: `func_dev`, `scale_dev`, `perf_dev`, `integ_dev` |
| 114 | 2. ``staging`` pod: `func_staging`, `scale_staging`, `perf_staging`, `integ_staging` |
| 115 | 3. ``qa`` pod: `func_qa`, `scale_qa`, `perf_qa`, `integ_qa` |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 116 | |
| 117 | Job structure |
| 118 | ^^^^^^^^^^^^^ |
| 119 | |
| 120 | Take `scale_dev` job as an example. It runs the following downstream jobs: |
| 121 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 122 | 1. `omec_deploy_dev`: this job re-deploys the dev pod with latest OMEC images. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 123 | |
| 124 | .. Note:: |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 125 | only the dev pod job triggers a deployment downstream job. No |
| 126 | re-deployment is performed on the staging and qa pod before the tests |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 127 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 128 | 2. `ng40-test_dev`: this job executes the scalability test suite. |
| 129 | 3. `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
| 130 | 4. `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the |
| 131 | test data to database. It also generates plots using Rscript for func and |
| 132 | scale tests |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 133 | |
| 134 | The integration tests are written using Robot Framework so have a slightly |
| 135 | different Jenkins Job structure. Take `integ_dev` as an example. It runs the |
| 136 | following downstream jobs: |
| 137 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 138 | 1. `omec_deploy_dev`: this job executes the scalability test suite. |
| 139 | 2. `robotframework-test_dev`: this job is similar to `ng40-test_dev` with the |
| 140 | exception that instead of directly executing NG40 commands it calls robot |
| 141 | framework to exectue the test cases and publishes the test results using |
| 142 | `RobotPublisher` Jenkins plugin. The robot results will also be copied to |
| 143 | the upstream job and published there. |
| 144 | 3. `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
| 145 | 4. `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the |
| 146 | test data to database. It also generates plots using Rscript for func and |
| 147 | scale tests |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 148 | |
| 149 | Patchset Tests |
| 150 | ~~~~~~~~~~~~~~ |
| 151 | |
| 152 | Overview |
| 153 | ^^^^^^^^ |
| 154 | |
| 155 | SD-Core pre-merge verifications cover the following Github repos: ``c3po``, |
| 156 | ``Nucleus``, ``upf-epc`` and ``spgw`` (private). OMEC CI includes the following |
| 157 | verifications: |
| 158 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 159 | 1. ONF CLA verification |
| 160 | 2. License verifications (FOSSA/Reuse) |
| 161 | 3. NG40 tests |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 162 | |
| 163 | These verifications are automatically triggered by submitted or updated PR to |
| 164 | the repos above. They can also be triggered manually by commenting ``retest |
| 165 | this please`` to the PR. At this moment only CLI and NG40 verifications are |
| 166 | mandatory. |
| 167 | |
| 168 | The NG40 verifications are a set of jobs running on both opencord Jenkins and |
| 169 | Aether Jenkins (private). The jobs run on opencord Jenkins include |
| 170 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 171 | 1. `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ (public) |
| 172 | 2. `omec_Nucleus_container_remote <https://jenkins.opencord.org/job/omec_Nucleus_container_remote/>`_ (public) |
| 173 | 3. `omec_upf-epc_container_remote <https://jenkins.opencord.org/job/omec_upf-epc_container_remote/>`_ (public) |
| 174 | 4. `omec_spgw_container_remote` (private, under member-only folder) |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 175 | |
| 176 | And the jobs run on Aether Jenkins include |
| 177 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 178 | 1. `c3po_premerge_dev` |
| 179 | 2. `Nucleus_premerge_dev` |
| 180 | 3. `upf-epc_premerge_dev` |
| 181 | 4. `spgw_premerge_dev` |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 182 | |
| 183 | Job structure |
| 184 | ^^^^^^^^^^^^^ |
| 185 | |
| 186 | Take c3po jobs as an example. c3po PR triggers a public job `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__ |
| 187 | job running on opencord Jenkins through Github webhooks, |
| 188 | which then triggers a private job `c3po_premerge_dev` running on Aether Jenkins |
| 189 | using a Jenkins plugin called `Parameterized Remote Trigger Plugin <https://www.jenkins.io/doc/pipeline/steps/Parameterized-Remote-Trigger/>`__. |
| 190 | |
| 191 | The private c3po job runs the following downstream jobs sequentially: |
| 192 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 193 | 1. `docker-publish-github_c3po`: this job downloads the c3po PR, runs docker |
| 194 | build and publishes the c3po docker images to `Aether registry`. |
| 195 | 2. `omec_deploy_dev`: this job deploys the images built from previous job onto |
| 196 | the omec dev pod. |
| 197 | 3. `ng40-test_dev`: this job executes the functionality test suite. |
| 198 | 4. `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 199 | |
| 200 | After all the downstream jobs are finished, the upstream job (`c3po_premerge_dev`) |
| 201 | copies artifacts including k8s/container/NG40 logs and pcap files from |
| 202 | downstream jobs and saves them as Jenkins job artifacts. |
| 203 | |
| 204 | These artifacts are also copied to and published by the public job |
| 205 | (`omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__) |
| 206 | on opencord Jenkins so that they can be accessed by the OMEC community. |
| 207 | |
| 208 | Pre-merge jobs for other OMEC repos share the same structure. |
| 209 | |
| 210 | Post-merge |
| 211 | ^^^^^^^^^^ |
| 212 | |
| 213 | The following jobs are triggered as post-merge jobs when PRs are merged to |
| 214 | OMEC repos: |
| 215 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 216 | 1. `docker-publish-github-merge_c3po` |
| 217 | 2. `docker-publish-github-merge_Nucleus` |
| 218 | 3. `docker-publish-github-merge_upf-epc` |
| 219 | 4. `docker-publish-github-merge_spgw` |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 220 | |
| 221 | Again take the c3po job as an example. The post-merge job (`docker-publish-github-merge_c3po`) |
| 222 | runs the following downstream jobs sequentially: |
| 223 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 224 | 1. `docker-publish-github_c3po`: this is the same job as the one in pre-merge |
| 225 | section. It checks out the latest c3po code, runs docker build and |
| 226 | publishes the c3po docker images to `docker hub <https://hub.docker.com/u/omecproject>`__. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 227 | |
| 228 | .. Note:: |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 229 | the spgw images are published to Aether registry instead of docker hub |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 230 | |
You Wang | aa55885 | 2021-04-15 15:41:54 -0700 | [diff] [blame] | 231 | 2. `c3po_postrelease`: this job submits a patchset to aether-pod-configs repo |
| 232 | for updating the CD pipeline with images published in the job above. |
You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 233 | |
| 234 | Post-merge jobs for other OMEC repos share the same structure. |