blob: cd40c4af6df1b5499520ec89ad1710e1acf2694a [file] [log] [blame]
Hardik Windlass1808a712022-03-10 11:23:30 +00001Voltha Test Automation
2======================
3
Joey Armstrong2bc25ac2024-04-19 12:21:42 -04004This page shows how the Voltha tests are automated to run on
5`OpenCORD Jenkins <https://jenkins.opencord.org/>`_.
Hardik Windlass1808a712022-03-10 11:23:30 +00006
Joey Armstrong2bc25ac2024-04-19 12:21:42 -04007Tests of different workflows, i.e. ATT, DT and TT, with various
8configurations based on either BBSim deployment or actual Physical POD run
9on Jenkins periodically.
Hardik Windlass1808a712022-03-10 11:23:30 +000010
11.. figure:: ../_static/Voltha_Test_Automation_Jenkins.png
12 :alt: Voltha OpenCord Jenkins
13 :width: 70%
14 :align: center
15
16 Voltha OpenCord Jenkins
17
18Similar set of jobs are kept under similar tabs. Like,
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040019all the master build jobs are placed under
20`VOLTHA-2.X-Tests <https://jenkins.opencord.org/view/VOLTHA-2.X-Tests/>`_
21tab, jobs related to the voltha-2.12 LTS release are in VOLTHA-2.12 tab.
Hardik Windlass1808a712022-03-10 11:23:30 +000022
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040023Likewise, tests that collect measurements on Voltha performance at scale
24are placed under `voltha-scale-measurements <https://jenkins.opencord.org/view/voltha-scale-measurements/>`_.
Hardik Windlass1808a712022-03-10 11:23:30 +000025
26Jenkins also host `verification jobs <https://jenkins.opencord.org/view/VOLTHA%202.x%20verify/>`_ that trigger automatically whenever a patch is pushed or updated on Gerrit for the review.
27
28
29Job Templates
30^^^^^^^^^^^^^
31
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040032Templates for the jobs are defined in
33`ci-management <https://github.com/opencord/ci-management>`__ repo.
Hardik Windlass1808a712022-03-10 11:23:30 +000034
35
36BBSim Jobs
37^^^^^^^^^^
38
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040039OpenCORD Jenkins host a large number of BBSim based regression jobs that
40run periodically with different deployment configurations to verify Voltha
41control plane functionality for different operator workflows. Such as,
Hardik Windlass1808a712022-03-10 11:23:30 +000042
43- `periodic-voltha-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-test-bbsim/>`_ & `periodic-voltha-multiple-olts-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-test-bbsim/>`_ : tests Voltha functional scenarios, recovery and error scenarios, ONOS HA scenarios etc. for ATT
44- `periodic-voltha-multi-uni-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multi-uni-test-bbsim/>`_ & `periodic-voltha-multi-uni-multiple-olts-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multi-uni-multiple-olts-test-bbsim/>`_ : tests Voltha functionality with multiple UNIs enabled, and with single & multiple OLT (BBSim) deployments
Joey Armstrongc5427b22023-08-15 08:55:33 -040045- `periodic-voltha-dt-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-dt-test-bbsim-master>`_ : tests functional and failure scenarios for DT workflow
Joey Armstrong449ce7a2023-07-18 18:32:24 -040046- `periodic-voltha-dt-fttb-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-dt-fttb-test-bbsim-master/>`_ : tests functional scenarios for DT FTTB workflow
Hardik Windlass1808a712022-03-10 11:23:30 +000047- `periodic-voltha-unitag-subscriber-tt-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-unitag-subscriber-tt-test-bbsim/>`_ : tests Voltha functionality with UNI-Tag Subscriber provisioning enabled for TT workflow
48- `periodic-voltha-tt-maclearner-sanity-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-tt-maclearner-sanity-test-bbsim/>`_ : tests Voltha functionality with MAC Learner enabled for TT workflow
49- `periodic-software-upgrade-test-bbsim <https://jenkins.opencord.org/job/periodic-software-upgrade-test-bbsim/>`_ : tests ONOS Applications, Voltha Components and ONU Software Upgrades
50- `periodic-voltha-openonu-go-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-openonu-go-test-bbsim/>`_ & `periodic-voltha-multiple-olts-openonu-go-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-openonu-go-test-bbsim/>`_ : tests state for ONU Go adapter with different operator workflows, different constellations and tech profiles
51- `periodic-voltha-pm-data-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-pm-data-test-bbsim/>`_ & `periodic-voltha-multiple-olts-pm-data-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-multiple-olts-pm-data-test-bbsim/>`_ : tests Open ONU Go adapter PM data
52- `periodic-voltha-etcd-test <https://jenkins.opencord.org/job/periodic-voltha-etcd-test/>`_ : tests Voltha etcd functionality
53- `periodic-voltha-test-DMI <https://jenkins.opencord.org/job/periodic-voltha-test-DMI/>`_ : tests device management interface functionality
Hardik Windlass9d8e6c82022-03-23 08:28:43 +000054- `periodic-voltha-memory-leak-test-bbsim <https://jenkins.opencord.org/job/periodic-voltha-memory-leak-test-bbsim/>`_ : tests and catches memory leak in Voltha components
Hardik Windlass1808a712022-03-10 11:23:30 +000055
56Similar to the above mentioned jobs which are based on **master**,
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040057Jenkins has regression jobs for other Voltha releases as well, like
58votlha-2.11 and voltha-2.12.
Hardik Windlass1808a712022-03-10 11:23:30 +000059
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040060The template for the jobs that run on BBSim based deployment is defined
61`here <https://github.com/opencord/ci-management/blob/master/jjb/voltha-e2e.yaml>`__.
Hardik Windlass1808a712022-03-10 11:23:30 +000062
63BBSim based deployment is also used to test Voltha at scale.
64The scale jobs run the tests that collect measurements on Voltha performance.
65
66Scale jobs do the deployment with various combinations
67like single or multiple voltha stacks with one or more BBSim based OLTs,
68multiple PONs and ONUs thus testing 4000+ subscribers for different workflows.
69More details about scale jobs can be found `here <https://jenkins.opencord.org/view/voltha-scale-measurements/>`__.
70
71
72Hardware (or Physical POD) Jobs
73^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
74
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040075OpenCord Jenkins host various regression jobs that runs on the ONF physical
76pods situated at different geographical locations. Hardware jobs test both
77control and dataplane functionality of Voltha.
Hardik Windlass1808a712022-03-10 11:23:30 +000078
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040079These hardware jobs are mostly defined in a combination of a build job and
80a test job. The template for the jobs that run on physical pod is defined
81`here <https://github.com/opencord/ci-management/blob/master/jjb/voltha-test/voltha-nightly-jobs.yaml>`__.
Hardik Windlass1808a712022-03-10 11:23:30 +000082
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040083Some of the configurations that are currently being tested on ONF's
84Physical Pods are:
Hardik Windlass1808a712022-03-10 11:23:30 +000085
86- Default Technology Profile with master or LTS release build for ATT
87- 1T4GEM profile with master or LTS release build for ATT
88- 1T4GEM-unencrypted profile with master or LTS release build for ATT
89- 1T8GEM Technology Profile with master or LTS release build for DT
90- 1T8GEM Technology Profile with master or LTS release build for DT with GPON based OLT setup
91- 1T8GEM Technology Profile with master or LTS release build for DT with multiple OLTs setup
92- Different service based Technology Profiles with master or LTS release build for TT
93- Different service based profiles with master or LTS release build for TT with multiple UNIs enabled
94
95
96Per-Patchset Verification Jobs
97^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
98
Joey Armstrong2bc25ac2024-04-19 12:21:42 -040099The verification jobs gets triggered automatically whenever a patch is
100pushed or updated on Gerrit, e.g.
Hardik Windlass1808a712022-03-10 11:23:30 +0000101
Joey Armstrong2bc25ac2024-04-19 12:21:42 -0400102- verify_{project}_licensed : This job verifies that there is valid
103 license/copyright on files.
104- verify_{project}_tag-collision : This job checks for changes to version
105 files, and that they don't duplicate tags already in the git repo.
Hardik Windlass1808a712022-03-10 11:23:30 +0000106- verify_{project}_unit-test : This job runs the Makefile based unit tests.
Joey Armstrong2bc25ac2024-04-19 12:21:42 -0400107- verify_{project}_sanity-test : This job verifies that basic Voltha
108 functionality is not broken for any of the operator workflows.
Hardik Windlass1808a712022-03-10 11:23:30 +0000109
Joey Armstrong2bc25ac2024-04-19 12:21:42 -0400110Apart from system triggered jobs, there are some verification jobs which
111can be triggered by the user by providing apt comment in the Gerrit patch,
112e.g.
Hardik Windlass1808a712022-03-10 11:23:30 +0000113
Joey Armstrong2bc25ac2024-04-19 12:21:42 -0400114- *hardware test functional tests* : This comment triggers the run on a
115 dedicated physical pod and executes the entire functional test suite for ATT.
Hardik Windlass1808a712022-03-10 11:23:30 +0000116
Joey Armstrong2bc25ac2024-04-19 12:21:42 -0400117- *hardware test functional tests DT* : This comment triggers the functional
118 suite tests for DT.
119- *voltha test openonu singleolt* :
120 Triggers patchset-voltha-openonu-go-test-bbsim job.
121- *voltha 2.9 test openonu singleolt* :
122 Triggers patchset-voltha-2.9-openonu-go-test-bbsim job.
123- *voltha 2.8 test openonu singleolt* :
124 Triggers patchset-voltha-2.8-openonu-go-test-bbsim job.
125- *voltha test openonu multiolt* :
126 Triggers patchset-voltha-multiple-olts-openonu-go-test-bbsim job.
127- *voltha 2.9 test openonu multiolt* :
128 Triggers patchset-voltha-2.9-multiple-olts-openonu-go-test-bbsim job.
129- *voltha 2.8 test openonu multiolt* :
130 Triggers patchset-voltha-2.8-multiple-olts-openonu-go-test-bbsim job.
131- *voltha test pm data singleolt* :
132 Triggers patchset-voltha-pm-data-test-bbsim job.
133- *voltha test pm data multiolt* :
134 Triggers patchset-voltha-multiple-olts-pm-data-test-bbsim job.
135
136The templates for the above mentioned and other similar jobs can be found
137in the `ci-management <https://github.com/opencord/ci-management>`__ repo.