You Wang | ee3a4db | 2021-04-13 18:50:40 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | SD-Core Testing |
| 6 | =============== |
| 7 | |
| 8 | Test Framework |
| 9 | -------------- |
| 10 | |
| 11 | NG40 |
| 12 | ~~~~ |
| 13 | |
| 14 | Overview |
| 15 | ^^^^^^^^ |
| 16 | |
| 17 | NG40 tool is used as RAN emulator in SD-Core testing. NG40 runs inside a VM |
| 18 | which is connected to both Aether control plane and data plane. In testing |
| 19 | scenarios that involve data plane verifications, NG40 also emulates a few |
| 20 | application servers which serve as the destinations of data packets. |
| 21 | |
| 22 | A typical NG40 test case involves UE attaching, data plane verifications and |
| 23 | UE detaching. During the test NG40 acts as UEs and eNBs and talks to the |
| 24 | mobile core to complete attach procedures for each UE it emulates. Then NG40 |
| 25 | verifies that data plane works for each attached UE by sending traffic between |
| 26 | UEs and application servers. Before finishing each test NG40 performs detach |
| 27 | procedures for each attached UE. |
| 28 | |
| 29 | Test cases |
| 30 | ^^^^^^^^^^ |
| 31 | |
| 32 | Currently the following NG40 test cases are implemented: |
| 33 | |
| 34 | 1. ``4G_M2AS_PING_FIX`` (attach, dl ping, detach) |
| 35 | 2. ``4G_M2AS_UDP`` (attach, dl+ul udp traffic, detach) |
| 36 | 3. ``4G_M2AS_TCP`` (attach, relaese, service request, dl+ul tcp traffic, detach) |
| 37 | 4. ``4G_AS2M_PAGING`` (attach, release, dl udp traffic, detach) |
| 38 | 5. ``4G_M2AS_SRQ_UDP`` (attach, release, service request, dl+ul udp traffic) |
| 39 | 6. ``4G_M2CN_PS`` (combined IMSI/PTMSI attach, detach) |
| 40 | 7. ``4G_HO`` (attach, relocate and ping, detach) |
| 41 | 8. ``4G_SCALE`` (attach with multiple UEs, ping, detach) |
| 42 | |
| 43 | All the test cases are parameterized and can take arguments to specify number |
| 44 | of UEs, attach/detach rate, traffic type/rate etc. For example, ``4G_SCALE`` |
| 45 | test case can be configured as a mini scalability test which performs only 5 |
| 46 | UE attaches in a patchset pre-merge test, while in the nightly tests it can |
| 47 | take different arguments to run 10K UE attaches with a high attach rate. |
| 48 | |
| 49 | Test suites |
| 50 | ^^^^^^^^^^^ |
| 51 | |
| 52 | The test cases are atomic testing units and can be combined to build test |
| 53 | suites. The following test suites have been built so far: |
| 54 | |
| 55 | * ``functionality test suite`` verifies basic functionality of the |
| 56 | mobile core. It runs test case #1 to #8 including ``4G_SCALE`` which attaches |
| 57 | 5 UEs with 1/s attach rate |
| 58 | |
| 59 | * ``scalability test suite`` tests the system by scale and verifies |
| 60 | system stability. It runs ``4G_SCALE`` which attaches a large number of UEs |
| 61 | with high attach rate (16k UEs with 100/s rate on dev pod, and 10k UEs with |
| 62 | 10/s rate on staging pod) |
| 63 | |
| 64 | * ``performance test suite`` measures performance of the control and |
| 65 | data plane. It runs ``4G_SCALE`` multiple times with different attach rates |
| 66 | to understand how the system performs under different loads. |
| 67 | |
| 68 | Robot Framework |
| 69 | ~~~~~~~~~~~~~~~ |
| 70 | |
| 71 | Robot Framework was chosen to build test cases that involve interacting with |
| 72 | not only NG40 but also other parts of the system. In these scenarios Robot |
| 73 | Framework acts as a high level orchestrator which drives various components |
| 74 | of the system using component specific libraries including NG40. |
| 75 | |
| 76 | Currently the ``Integration test suite`` is implemented using Robot |
| 77 | Framework. In the integration tests Robot Framework calls ng40 library to |
| 78 | perform normal attach/detach procedures. Meanwhile it injects failures into |
| 79 | the system (container restarts, link down etc.) by calling functions |
| 80 | implemented in the k8s library. |
| 81 | |
| 82 | The following integration tests are implemented at the moment: |
| 83 | |
| 84 | * Subscriber Attach with HSS Restart |
| 85 | * Subscriber Attach with MME Restart |
| 86 | * Subscriber Attach with SPGWC Restart |
| 87 | * Subscriber Attach with PFCP Agent Restart |
| 88 | |
| 89 | .. Note:: |
| 90 | More integration tests are being developed as part of Robot Framework |
| 91 | |
| 92 | Test Schedules |
| 93 | -------------- |
| 94 | |
| 95 | Nightly Tests |
| 96 | ~~~~~~~~~~~~~ |
| 97 | |
| 98 | Overview |
| 99 | ^^^^^^^^ |
| 100 | |
| 101 | SD-Core nightly tests are a set of jobs managed by Aether Jenkins. |
| 102 | All four test suites we mentioned above are scheduled to run nightly. |
| 103 | |
| 104 | * ``functionality job (func)`` runs NG40 test cases included in the |
| 105 | functionality suite and verifies all tests pass. |
| 106 | |
| 107 | * ``scalability job (scale)`` runs the scalability test suite and reports |
| 108 | the number of successful/failed attaches, detaches and pings. |
| 109 | |
| 110 | * ``performance job (perf)`` runs the performance test suite and reports |
| 111 | SCTP heartbeat RTT, GTP ICMP RTT and call setup latency numbers. |
| 112 | |
| 113 | And all these jobs can be scheduled on any of the Aether PODs including |
| 114 | ``dev`` pod, ``staging`` pod and ``qa`` pod. By combining the test type and |
| 115 | test pod the following Jenkins jobs are generated: |
| 116 | |
| 117 | * ``dev`` pod: `func_dev`, `scale_dev`, `perf_dev`, `integ_dev` |
| 118 | * ``staging`` pod: `func_staging`, `scale_staging`, `perf_staging`, `integ_staging` |
| 119 | * ``qa`` pod: `func_qa`, `scale_qa`, `perf_qa`, `integ_qa` |
| 120 | |
| 121 | Job structure |
| 122 | ^^^^^^^^^^^^^ |
| 123 | |
| 124 | Take `scale_dev` job as an example. It runs the following downstream jobs: |
| 125 | |
| 126 | * `omec_deploy_dev`: this job re-deploys the dev pod with latest OMEC images. |
| 127 | |
| 128 | .. Note:: |
| 129 | only the dev pod job triggers a deployment downstream job. No |
| 130 | re-deployment is performed on the staging and qa pod before the tests |
| 131 | |
| 132 | * `ng40-test_dev`: this job executes the scalability test suite. |
| 133 | |
| 134 | * `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
| 135 | |
| 136 | * `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the |
| 137 | test data to database. It also generates plots using Rscript for func and |
| 138 | scale tests |
| 139 | |
| 140 | The integration tests are written using Robot Framework so have a slightly |
| 141 | different Jenkins Job structure. Take `integ_dev` as an example. It runs the |
| 142 | following downstream jobs: |
| 143 | |
| 144 | * `omec_deploy_dev`: this job executes the scalability test suite. |
| 145 | |
| 146 | * `robotframework-test_dev`: this job is similar to `ng40-test_dev` with the |
| 147 | exception that instead of directly executing NG40 commands it calls robot |
| 148 | framework to exectue the test cases and publishes the test results using |
| 149 | `RobotPublisher` Jenkins plugin. The robot results will also be copied to |
| 150 | the upstream job and published there. |
| 151 | |
| 152 | * `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
| 153 | |
| 154 | * `post-results_dev`: this job collects the NG40 test logs/pcaps and pushes the |
| 155 | test data to database. It also generates plots using Rscript for func and |
| 156 | scale tests |
| 157 | |
| 158 | Patchset Tests |
| 159 | ~~~~~~~~~~~~~~ |
| 160 | |
| 161 | Overview |
| 162 | ^^^^^^^^ |
| 163 | |
| 164 | SD-Core pre-merge verifications cover the following Github repos: ``c3po``, |
| 165 | ``Nucleus``, ``upf-epc`` and ``spgw`` (private). OMEC CI includes the following |
| 166 | verifications: |
| 167 | |
| 168 | * ONF CLA verification |
| 169 | * License verifications (FOSSA/Reuse) |
| 170 | * NG40 tests |
| 171 | |
| 172 | These verifications are automatically triggered by submitted or updated PR to |
| 173 | the repos above. They can also be triggered manually by commenting ``retest |
| 174 | this please`` to the PR. At this moment only CLI and NG40 verifications are |
| 175 | mandatory. |
| 176 | |
| 177 | The NG40 verifications are a set of jobs running on both opencord Jenkins and |
| 178 | Aether Jenkins (private). The jobs run on opencord Jenkins include |
| 179 | |
| 180 | * `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ (public) |
| 181 | |
| 182 | * `omec_Nucleus_container_remote <https://jenkins.opencord.org/job/omec_Nucleus_container_remote/>`_ (public) |
| 183 | |
| 184 | * `omec_upf-epc_container_remote <https://jenkins.opencord.org/job/omec_upf-epc_container_remote/>`_ (public) |
| 185 | |
| 186 | * `omec_spgw_container_remote` (private, under member-only folder) |
| 187 | |
| 188 | And the jobs run on Aether Jenkins include |
| 189 | |
| 190 | * `c3po_premerge_dev` |
| 191 | |
| 192 | * `Nucleus_premerge_dev` |
| 193 | |
| 194 | * `upf-epc_premerge_dev` |
| 195 | |
| 196 | * `spgw_premerge_dev` |
| 197 | |
| 198 | Job structure |
| 199 | ^^^^^^^^^^^^^ |
| 200 | |
| 201 | Take c3po jobs as an example. c3po PR triggers a public job `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__ |
| 202 | job running on opencord Jenkins through Github webhooks, |
| 203 | which then triggers a private job `c3po_premerge_dev` running on Aether Jenkins |
| 204 | using a Jenkins plugin called `Parameterized Remote Trigger Plugin <https://www.jenkins.io/doc/pipeline/steps/Parameterized-Remote-Trigger/>`__. |
| 205 | |
| 206 | The private c3po job runs the following downstream jobs sequentially: |
| 207 | |
| 208 | * `docker-publish-github_c3po`: this job downloads the c3po PR, runs docker |
| 209 | build and publishes the c3po docker images to `Aether registry`. |
| 210 | |
| 211 | * `omec_deploy_dev`: this job deploys the images built from previous job onto |
| 212 | the omec dev pod. |
| 213 | |
| 214 | * `ng40-test_dev`: this job executes the functionality test suite. |
| 215 | |
| 216 | * `archive-artifacts_dev`: this job collects and uploads k8s and container logs. |
| 217 | |
| 218 | After all the downstream jobs are finished, the upstream job (`c3po_premerge_dev`) |
| 219 | copies artifacts including k8s/container/NG40 logs and pcap files from |
| 220 | downstream jobs and saves them as Jenkins job artifacts. |
| 221 | |
| 222 | These artifacts are also copied to and published by the public job |
| 223 | (`omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`__) |
| 224 | on opencord Jenkins so that they can be accessed by the OMEC community. |
| 225 | |
| 226 | Pre-merge jobs for other OMEC repos share the same structure. |
| 227 | |
| 228 | Post-merge |
| 229 | ^^^^^^^^^^ |
| 230 | |
| 231 | The following jobs are triggered as post-merge jobs when PRs are merged to |
| 232 | OMEC repos: |
| 233 | |
| 234 | * `docker-publish-github-merge_c3po` |
| 235 | |
| 236 | * `docker-publish-github-merge_Nucleus` |
| 237 | |
| 238 | * `docker-publish-github-merge_upf-epc` |
| 239 | |
| 240 | * `docker-publish-github-merge_spgw` |
| 241 | |
| 242 | Again take the c3po job as an example. The post-merge job (`docker-publish-github-merge_c3po`) |
| 243 | runs the following downstream jobs sequentially: |
| 244 | |
| 245 | * `docker-publish-github_c3po`: this is the same job as the one in pre-merge |
| 246 | section. It checks out the latest c3po code, runs docker build and |
| 247 | publishes the c3po docker images to `docker hub <https://hub.docker.com/u/omecproject>`__. |
| 248 | |
| 249 | .. Note:: |
| 250 | the spgw images are published to Aether registry instead of docker hub |
| 251 | |
| 252 | * `c3po_postrelease`: this job submits a patchset to aether-pod-configs repo |
| 253 | for updating the CD pipeline with images published in the job above. |
| 254 | |
| 255 | Post-merge jobs for other OMEC repos share the same structure. |