Updated Testing section
Change-Id: I6e7bf4b85100b60e8e24ed6c0b65cc19b961f4ba
diff --git a/dict.txt b/dict.txt
index 44cd455..bd46000 100644
--- a/dict.txt
+++ b/dict.txt
@@ -1,3 +1,24 @@
+Rscript
+orchestrator
+Grafana
+eNBID
+macroENB
+webhooks
+func
+scalability
+Speedtest
+SupportedTAs
+pcaps
+pcap
+tcp
+deregistration
+dl
+ul
+ng
+qa
+Github
+LTE
+Wireshark
disaggregation
natively
onpremises
diff --git a/index.rst b/index.rst
index c38bc42..892631e 100644
--- a/index.rst
+++ b/index.rst
@@ -70,6 +70,17 @@
deployment/deployment5G.rst
.. toctree::
+ :maxdepth: 3
+ :caption: SD-Core Test Automation
+ :hidden:
+ :glob:
+
+ testing/about_system_tests
+ testing/sdcore_testing
+ testing/acceptance_specification
+
+
+.. toctree::
:maxdepth: 2
:caption: Troubleshooting Guide
:hidden:
diff --git a/testing/about_system_tests.rst b/testing/about_system_tests.rst
new file mode 100644
index 0000000..b1ec5de
--- /dev/null
+++ b/testing/about_system_tests.rst
@@ -0,0 +1,41 @@
+..
+ SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org>
+ SPDX-License-Identifier: Apache-2.0
+
+About
+=====
+
+The goal and objective of SD-Core test automation is to build a framework that
+provides highly scalable and low maintenance code which will help cover various
+categories of tests. Framework includes libraries and tools that allows both
+component level and integration level tests. Robot Framework will be used for
+covering integration tests. Component level test coverage have been
+accomplished by leveraging the existing test frameworks that were developed in
+their respective projects. Component level tests include tests for SD-CORE areas.
+
+Test Frameworks
+---------------
+
+The following diagram outlines each SD-Core component, followed by an online
+of the test frameworks used:
+
+.. image:: images/4G-Common-Testing.png
+ :width: 840
+ :height: 540
+ :alt: Aether Overview Diagram
+
+.. list-table:: Test Frameworks
+ :widths: 5 3
+ :header-rows: 1
+
+ * - Aether Core Component
+ - Test Framework
+ * - SD-Core
+ - `NG40/NG4T <https://www.ng4t.com/>`_
+
+Test Automation
+---------------
+
+`Jenkins <https://www.jenkins.io/>`_ is the primary automation server that is
+used to help trigger our automated tests. All Aether Jenkins jobs are
+created and run on the Aether Jenkins instance.
diff --git a/testing/acceptance_specification.rst b/testing/acceptance_specification.rst
new file mode 100644
index 0000000..e1c8482
--- /dev/null
+++ b/testing/acceptance_specification.rst
@@ -0,0 +1,332 @@
+..
+ SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
+ SPDX-License-Identifier: Apache-2.0
+
+Acceptance Specification
+========================
+
+Objectives
+----------
+
+The purpose of this document is to create an end-user test object list (TOL)
+for Aether Connected Edge (ACE).
+
+This document will focus on the connectivity services end-user testing. In the
+future, this document will extend to other services offered through ACE.
+
+The Automated continuous testing framework for the platform is out of the scope of this document.
+
+Integration Test (eNB-LTE Core)
+-------------------------------
+
+Before we start to test End-to-End connectivity, we have to check the
+connection (called S1-MME/S1-C interface) between eNB in an edge and MME in a
+public cloud.
+
+In order to verify this connectivity, the following test cases should be passed.
+
+Note that all the following test/verification cases have some assumptions:
+
+1. eNB is connected to the Fabric switch;
+2. eNB has correct network configurations;
+3. eNB has correct ID configurations provided by the ONF PMFE team.
+
+IT-TOL01 Fabric Test 1: the connectivity test within the edge
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to test the fabric test, please see the following steps:
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Access to the eNB* (through SSH or Web GUI)|Able to get ICMP reply(PING reply) |
+| |from the GCP node. |
+| | |
+|2. Ping to "10.168.0.6" eNB S1-MME/S1-C |( )YES ( )NO |
+| interface IP address* | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+.. note::
+ it depends on the eNB device. Some eNBs have a single network interface for the management network, S1-U, and S1-C,
+ while other eNBs have separate interfaces. The former eNB type has a single IP address,
+ and the later eNB type has multiple IP addresses for each interface.
+
+IT-TOL02 Fabric Test 2: the connectivity test between the eNB and the public cloud
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to test the fabric test, please see the following steps:
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Access to the eNB* (through SSH or Web GUI)|Able to get ICMP reply(PING reply) |
+| |from the GCP node. |
+| | |
+|2. Ping to "10.168.0.6" |( )YES ( )NO |
+| | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+
+
+.. note::
+ it also depends on the eNB device. Some eNBs allow us to SSH into their device, other eNBs provide the PING tool through Web GUI.
+ Of course, some eNBs do not support both. In that case, it is okay to skip this test case.
+
+
+IT-TOL03 SCTP Connection Verification
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to verify the SCTP connection between MME and eNB, please see the following steps:
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. SSH to the gateway device* for the eNB |Able to see Heart Beat |
+| S1-MME/S1-C traffic |messages |
+| | |
+|2. Capture the traffic with Wireshark |( )YES ()NO |
+| or the command: `sudo tcpdump -i any sctp` | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+Capture the traffic with Wireshark or the command: `sudo tcpdump -i any sctp`
+
+.. note::
+ the eNB should have the gateway IP address for the S1-MME/S1-C traffic.
+ You can SSH there with the gateway IP address in the eNB and capture the traffic.
+ Normally, the gateway device can be one of those devices: (i) VPN router; (ii) Firewall device in between VPN router and the edge;
+ (iii) one of edge servers.
+
+IT-TOL04 PLMN Verification
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to verify the TAC number, please see the following steps:
+
+
+Able to see the correct MCC and MNC values*
+
++-------------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++-------------------------------------------------+------------------------------------+
+|1. SSH to the gateway device for the eNB |Able to see the correct MCC and MNC |
+| S1-MME/S1-C traffic |values |
+| | |
+|2. Start to capture the traffic with Wireshark |( )YES ()NO |
+| or the command: `sudo tcpdump -i any sctp | |
+| -w FileName.pcap` |Comments: |
+|3. Reboot eNB | |
+| | |
+|4. Wait until FileName.pcap has `S1SetupRequest` | |
+| S1SetupResponse messages | |
+| | |
+|5. Stop the packet capturing and open | |
+| the FileName.pcap | |
+| | |
+|6. Find out the S1SetupRequest message and | |
+| open the detailed packet information | |
+| | |
+|7. Go to "Item 2: id-SupportedTAs" section | |
+| and check "MACC and "MNC" values | |
++-------------------------------------------------+------------------------------------+
+
+Example (the MCC is 315 and MNC is 010)
+
+.. image:: images/it-tol04.png
+ :width: 840
+ :height: 840
+ :alt: Example (the MCC is 315 and MNC is 010)
+
+IT-TOL05 TAC Number Verification
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
++-------------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++-------------------------------------------------+------------------------------------+
+|1. SSH to the gateway device for the eNB |Able to see the correct TAC number |
+| S1-MME/S1-C traffic | |
+| | |
+|2. Start to capture the traffic with Wireshark |( )YES ()NO |
+| or the command: `sudo tcpdump -i any sctp | |
+| -w FileName.pcap` |Comments: |
+|3. Reboot eNB | |
+| | |
+|4. Wait until FileName.pcap has `S1SetupRequest` | |
+| S1SetupResponse messages | |
+| | |
+|5. Stop the packet capturing and open | |
+| the FileName.pcap | |
+| | |
+|6. Find out the S1SetupRequest message and | |
+| open the detailed packet information | |
+| | |
+|7. Go to "Item 0: id-SupportedTAs" section | |
+| and check TAC " | |
++-------------------------------------------------+------------------------------------+
+
+.. note::
+ if you already captured packets in IT-TOL03, you can skip steps from 1 to 5.
+ Just you can check the expected outcome with the file you captured at IT-TOL03.
+
+Example (the TAC number is 19)
+
+.. image:: images/it-tol05.png
+ :width: 840
+ :height: 840
+ :alt: Example (the TAC number is 19)
+
+IT-TOL06 eNB Verification
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to test the eNB, please see the following steps:
+
++-------------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++-------------------------------------------------+------------------------------------+
+|1. SSH to the gateway device for the eNB |Able to see the correct eNBID |
+| S1-MME/S1-C traffic | |
+| | |
+|2. Start to capture the traffic with Wireshark |( )YES ()NO |
+| or the command: `sudo tcpdump -i any sctp | |
+| -w FileName.pcap` |Comments: |
+|3. Reboot eNB | |
+| | |
+|4. Wait until FileName.pcap has `S1SetupRequest` | |
+| S1SetupResponse messages | |
+| | |
+|5. Stop the packet capturing and open | |
+| the FileName.pcap | |
+| | |
+|6. Find out the S1SetupRequest message and | |
+| open the detailed packet information | |
+| | |
+|7. Go to "Item 0: id-Global-eNB-ID" section | |
+| and check "eNB-ID: macroENB-ID" | |
++-------------------------------------------------+------------------------------------+
+
+.. note::
+ if you already captured packets in IT-TOL03, you can skip steps number 1 to 5.
+ Just you can check the expected outcome with the file you captured at IT-TOL03.
+
+Example (the eNB ID is 19)
+
+.. image:: images/it-tol06.png
+ :width: 840
+ :height: 840
+ :alt: Example (the eNB ID is 19)
+
+Connectivity Services
+---------------------
+
+Aether provides only data connectivity for end-user devices and systems.
+So the voice service over LTE is not available. However, users can use
+any OTT services over the Aether network for voice connectivity.
+
+The test specifications are only covering the data connectivity focused tests.
+
+
+CS-TOL01 Device Attach/Connect
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To test device can attach to Aether network
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Turn off the mobile device |Able to attach the device and |
+| |connect to the internet/Aether |
+|2. Turn on the mobile device |Network |
+| | |
+|3. Check whether the device is showing |( )YES ( )NO |
+| connected on the status, depending on | |
+| the device it will show "Aether" or | |
+| "MCCMNC" format. | |
+|4. Browse http://www.google.com/? |( )YES ( )NO |
+| From the device web browser | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+CS-TOL02 Device Detach/Disconnect
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To test device can detach/disconnected by user initiation
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Make sure the device is connected to Aether|Able to detach the device and |
+| |disconnect from the internet/Aether |
+|2. Deselect the network (or forget the network|Network |
+| , depending on device configuration) | |
+|3. Try to browse http://www.google.com/? |( )YES ( )NO |
+| From your web browser | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+
+CS-TOL03 Bandwidth Test - Internet
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To test bandwidth available to a mobile device over Aether network.
+
+Please note the following, the bandwidth test depends on the eNB hardware,
+your local breakout bandwidth, and the overall radio environment.
+If you face an unexpected result, please explain it in the comment section in the outcome column.
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Open Speedtest app from your mobile device |Expected Bandwidth/Throughput |
+| |observed |
+| | |
+|2. Run Speedtest 3 times, take the average as |( )YES ( )NO |
+| the final result | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+
+CS-TOL04 Bandwidth Test - Edge Application
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To test bandwidth available to a mobile device over Aether network.
+
+Please note the following, the bandwidth test depends on the eNB hardware,
+your local breakout bandwidth, and the overall radio environment. If you face an unexpected result,
+please explain it in the comment section in the outcome column.
+
+
++----------------------------------------------+------------------------------------+
+|Steps |Expected Outcome |
++----------------------------------------------+------------------------------------+
+|1. Initiate FTP Download from a local server |Expected Bandwidth/Throughput |
+| (same location) connected to the enterprise|observed |
+| network (through local breakout) | |
+| | |
+|2. Download 3 times, take the average as the |( )YES ( )NO |
+| final result | |
+| |Comments: |
++----------------------------------------------+------------------------------------+
+
+
+
+Monitoring Services
+-------------------
+
+ACE uses the Grafana dashboard for monitoring services.
+Each ACE will be provided with Read-Only Access to our centralized monitoring platform.
+
+
+Application Services
+--------------------
+
+Aether uses Rancher to onboard applications to ACE.
+Each ACE host will be provided with access to rancher to onboard applications on their ACE cluster.
+
diff --git a/testing/images/4G-Common-Testing.png b/testing/images/4G-Common-Testing.png
new file mode 100644
index 0000000..910e991
--- /dev/null
+++ b/testing/images/4G-Common-Testing.png
Binary files differ
diff --git a/testing/images/aether-overview-diagram.png b/testing/images/aether-overview-diagram.png
new file mode 100644
index 0000000..3b22271
--- /dev/null
+++ b/testing/images/aether-overview-diagram.png
Binary files differ
diff --git a/testing/images/it-tol04.png b/testing/images/it-tol04.png
new file mode 100644
index 0000000..0401420
--- /dev/null
+++ b/testing/images/it-tol04.png
Binary files differ
diff --git a/testing/images/it-tol05.png b/testing/images/it-tol05.png
new file mode 100644
index 0000000..a71812a
--- /dev/null
+++ b/testing/images/it-tol05.png
Binary files differ
diff --git a/testing/images/it-tol06.png b/testing/images/it-tol06.png
new file mode 100644
index 0000000..ac92662
--- /dev/null
+++ b/testing/images/it-tol06.png
Binary files differ
diff --git a/testing/sdcore_testing.rst b/testing/sdcore_testing.rst
new file mode 100644
index 0000000..ed3be94
--- /dev/null
+++ b/testing/sdcore_testing.rst
@@ -0,0 +1,264 @@
+..
+ SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org>
+ SPDX-License-Identifier: Apache-2.0
+
+SD-Core Testing
+===============
+
+Test Framework
+--------------
+
+NG40
+""""
+
+NG40 tool is used as RAN emulator in SD-Core testing. NG40 runs inside a VM
+which is connected to both Aether control plane and data plane. In testing
+scenarios that involve data plane verification, NG40 also emulates a few
+application servers which serve as the destinations of data packets.
+
+A typical NG40 test case involves UE attaching, data plane verification and
+UE detaching. During the test NG40 acts as UEs and eNBs and talks to the
+mobile core to complete attach procedures for each UE it emulates. Then NG40
+verifies that data plane works for each attached UE by sending traffic between
+UEs and application servers. Before finishing each test NG40 performs detach
+procedures for each attached UE.
+
+Test cases
+''''''''''
+
+Currently the following NG40 test cases are implemented:
+
+4G Tests:
+
+1. ``4G_M2AS_PING_FIX`` (attach, dl ping, detach)
+2. ``4G_M2AS_UDP`` (attach, dl+ul udp traffic, detach)
+3. ``4G_M2AS_TCP`` (attach, release, service request, dl+ul tcp traffic, detach)
+4. ``4G_AS2M_PAGING`` (attach, release, dl udp traffic, detach)
+5. ``4G_M2AS_SRQ_UDP`` (attach, release, service request, dl+ul udp traffic)
+6. ``4G_M2CN_PS`` (combined IMSI/PTMSI attach, detach)
+7. ``4G_HO`` (attach, relocate and dl ping, detach)
+8. ``4G_SCALE`` (attach, dl ping, detach with multiple UEs)
+
+5G Tests:
+
+1. ``5G_SA_Register_Deregister`` (registration, deregistration)
+2. ``5G_SA_Register`` (registration, session establishment, deregistration)
+3. ``5G_SA_Release`` (registration, session establishment, dl ping, release, deregistration)
+4. ``5G_SA_Activate_Release`` (registration, session establishment, dl ping, release, service request,
+ dl ping, deregistration)
+5. ``5G_SA_Scale`` (registration, session establishment, dl ping, deregistration for multiple UEs)
+6. ``5G_SA_M2AS_ICMP`` (registration, session establishment, dl ping, deregistration)
+7. ``5G_SA_M2AS_TCP`` (registration, session establishment, dl+ul tcp traffic, deregistration)
+8. ``5G_SA_M2AS_UDP`` (registration, session establishment, dl+ul udp traffic, deregistration)
+
+All the test cases are parameterized and can take arguments to specify number
+of UEs, attach/detach rate, traffic type/rate etc. For example, ``4G_SCALE``
+test case can be configured as a mini scalability test which performs only 5
+UE attaches in a patchset pre-merge test, while in the nightly tests it can
+take different arguments to run 10K UE attaches with a high attach rate.
+
+Test suites
+'''''''''''
+
+The test cases are atomic testing units and can be combined to build test
+suites. The following test suites have been built so far:
+
+1. ``functionality test suite`` verifies basic functionality of the
+ mobile core. 4G functionality suite runs 4G test case #1 to #8 including
+ ``4G_SCALE`` which attaches 5 UEs with 1/s attach rate. 5G functionality
+ suite runs 5G test case #1 to #8 including ``5G_SA_Scale`` which attaches
+ 100 UEs with 1/s attach rate.
+2. ``scalability test suite`` tests the system by scale and verifies
+ system stability. It runs ``4G_SCALE`` (or ``5G_SA_Scale``) which attaches
+ a large number of UEs with high attach rate (16k UEs with 100/s rate on 4G
+ CI pod, 1k UEs with 10/s rate on 4G staging pod, and 1k UEs with 1/s rate
+ on 5G CI pod).
+3. ``performance test suite`` measures performance of the control and
+ data plane. It runs ``4G_SCALE`` multiple times with different attach rates
+ to understand how the system performs under different loads.
+
+Robot Framework
+"""""""""""""""
+
+Robot Framework was chosen to build test cases that involve interacting with
+not only NG40 but also other parts of the system. In these scenarios Robot
+Framework acts as a high level orchestrator which drives various components
+of the system using component specific libraries including NG40.
+
+Currently the ``Integration test suite`` is implemented using Robot
+Framework. In the integration tests Robot Framework calls the ng40 library to
+perform normal attach/detach procedures. Meanwhile it injects failures into
+the system (container restarts, link down etc.) by calling functions
+implemented in the k8s library.
+
+The following integration tests are implemented at the moment:
+
+1. Subscriber Attach with HSS Restart
+2. Subscriber Attach with MME Restart
+3. Subscriber Attach with SPGWC Restart
+4. Subscriber Attach with PFCP Agent Restart
+
+.. Note::
+ More integration tests are being developed as part of Robot Framework
+
+Test Schedules
+--------------
+
+Nightly Tests
+"""""""""""""
+
+SD-Core nightly tests are a set of jobs managed by Aether Jenkins.
+All four test suites we mentioned above are scheduled to run nightly.
+
+1. ``functionality job (func)`` runs NG40 test cases included in the
+ functionality suite and verifies all tests pass.
+2. ``scalability job (scale)`` runs the scalability test suite and reports
+ the number of successful/failed attaches, detaches and pings.
+3. ``performance job (perf)`` runs the performance test suite and reports
+ SCTP heartbeat RTT, GTP ICMP RTT and call setup latency numbers.
+
+And all these jobs can be scheduled on any of the Aether PODs including
+``ci-4g`` pod, ``ci-5g`` pod, ``staging`` pod and ``qa`` pod. By combining
+the test type and test pod the following Jenkins jobs are generated:
+
+1. ``ci-4g`` pod: `sdcore_func_ci-4g`, `sdcore_scale_ci-4g`, `sdcore_perf_ci-4g`, `sdcore_integ_ci-4g`
+1. ``ci-5g`` pod: `sdcore_func_ci-5g`, `sdcore_scale_ci-5g`
+2. ``staging`` pod: `func_staging`, `scale_staging`, `perf_staging`, `integ_staging`
+3. ``qa`` pod: `func_qa`, `scale_qa`, `perf_qa`, `integ_qa`
+
+Nightly Job structure
+"""""""""""""""""""""
+
+Take `sdcore_scale_ci-4g` job as an example. It runs the following downstream jobs:
+
+1. `omec_deploy_ci-4g`: this job re-deploys the ``ci-4g`` pod with latest OMEC images.
+
+.. Note::
+ only the ``ci-4g`` and ``ci-5g`` pod jobs trigger deployment downstream job. No
+ re-deployment is performed on the staging and qa pod before the tests
+
+2. `ng40-test_ci-4g`: this job executes the scalability test suite.
+3. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
+4. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
+ test data to database. It also generates plots using Rscript for func and
+ scale tests
+
+The integration tests are written using Robot Framework so have a slightly
+different Jenkins Job structure. Take `sdcore_integ_ci-4g` as an example. It runs the
+following downstream jobs:
+
+1. `omec_deploy_ci-4g`: this job executes the scalability test suite.
+2. `robotframework-test_ci-4g`: this job is similar to `ng40-test_ci-4g` with the
+ exception that instead of directly executing NG40 commands it calls robot
+ framework to execute the test cases and publishes the test results using
+ `RobotPublisher` Jenkins plugin. The robot results will also be copied to
+ the upstream job and published there.
+3. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
+4. `post-results_ci-4g`: this job collects the NG40 test logs/pcaps and pushes the
+ test data to database. It also generates plots using Rscript for func and
+ scale tests
+
+Patchset Tests
+--------------
+
+SD-Core pre-merge verification covers the following public Github repos: ``c3po``,
+``Nucleus``, ``upf-epc`` and the following private Github repos: ``spgw``. ``amf``,
+``smf``, ``ausf``, ``nssf``, ``nrf``, ``pcf``, ``udm``, ``udr``, ``webconsole``.
+SD-Core CI verifies the following:
+
+1. ONF CLA verification
+2. License verification (FOSSA/Reuse)
+3. NG40 tests
+
+These jobs are automatically triggered by submitted or updated PR to the repos
+above. They can also be triggered manually by commenting ``retest this please``
+to the PR. At this moment only CLI and NG40 verification are mandatory.
+
+The NG40 verification are a set of jobs running on both opencord Jenkins and
+Aether Jenkins (private). The jobs run on opencord Jenkins include
+
+1. `omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ (public)
+2. `omec_Nucleus_container_remote <https://jenkins.opencord.org/job/omec_Nucleus_container_remote/>`_ (public)
+3. `omec_upf-epc_container_remote <https://jenkins.opencord.org/job/omec_upf-epc_container_remote/>`_ (public)
+4. `omec_spgw_container_remote` (private, under member-only folder)
+
+And the jobs run on Aether Jenkins include
+
+1. `c3po_premerge_ci-4g`
+2. `Nucleus_premerge_ci-4g`
+3. `upf-epc_premerge_ci-4g`
+4. `spgw_premerge_ci-4g`
+5. `amf_premerge_ci-5g`
+6. `smf_premerge_ci-5g`
+7. `ausf_premerge_ci-5g`
+8. `nssf_premerge_ci-5g`
+9. `nrf_premerge_ci-5g`
+10. `pcf_premerge_ci-5g`
+11. `udm_premerge_ci-5g`
+12. `udr_premerge_ci-5g`
+13. `webconsole_premerge_ci-5g`
+
+Patchset Job structure
+""""""""""""""""""""""
+
+Take ``c3po`` jobs as an example. ``c3po`` PR triggers a public job
+`omec_c3po_container_remote
+<https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_ job running
+on opencord Jenkins through Github webhooks, which then triggers a private job
+`c3po_premerge_ci-4g` running on Aether Jenkins using a Jenkins plugin called
+`Parameterized Remote Trigger Plugin
+<https://www.jenkins.io/doc/pipeline/steps/Parameterized-Remote-Trigger/>`_.
+
+The private ``c3po`` job runs the following downstream jobs sequentially:
+
+1. `docker-publish-github_c3po`: this job downloads the ``c3po`` PR, runs docker
+ build and publishes the ``c3po`` docker images to `Aether registry`.
+2. `omec_deploy_ci-4g`: this job deploys the images built from previous job onto
+ the omec ``ci-4g`` pod.
+3. `ng40-test_ci-4g`: this job executes the functionality test suite.
+4. `archive-artifacts_ci-4g`: this job collects and uploads k8s and container logs.
+
+After all the downstream jobs are finished, the upstream job (`c3po_premerge_ci-4g`)
+copies artifacts including k8s/container/ng40 logs and pcap files from
+downstream jobs and saves them as Jenkins job artifacts.
+
+These artifacts are also copied to and published by the public job
+(`omec_c3po_container_remote <https://jenkins.opencord.org/job/omec_c3po_container_remote/>`_)
+on opencord Jenkins so that they can be accessed by the OMEC community.
+
+Pre-merge jobs for other SD-Core repos share the same structure.
+
+Post-merge
+""""""""""
+
+The following jobs are triggered as post-merge jobs when PRs are merged to
+SD-Core repos:
+
+1. `docker-publish-github-merge_c3po`
+2. `docker-publish-github-merge_Nucleus`
+3. `docker-publish-github-merge_upf-epc`
+4. `docker-publish-github-merge_spgw`
+5. `docker-publish-github-merge_amf`
+6. `docker-publish-github-merge_smf`
+7. `docker-publish-github-merge_ausf`
+8. `docker-publish-github-merge_nssf`
+9. `docker-publish-github-merge_nrf`
+10. `docker-publish-github-merge_pcf`
+11. `docker-publish-github-merge_udm`
+12. `docker-publish-github-merge_udr`
+13. `docker-publish-github-merge_webconsole`
+
+Again take the ``c3po`` job as an example. The post-merge job (`docker-publish-github-merge_c3po`)
+runs the following downstream jobs sequentially:
+
+1. `docker-publish-github_c3po`: this is the same job as the one in pre-merge
+ section. It checks out the latest ``c3po`` code, runs docker build and
+ publishes the ``c3po`` docker images to `docker hub <https://hub.docker.com/u/omecproject>`__.
+
+.. Note::
+ the images for private repos are published to Aether registry instead of docker hub
+
+2. `c3po_postrelease`: this job submits a patchset to aether-pod-configs repo
+ for updating the CD pipeline with images published in the job above.
+
+Post-merge jobs for other SD-Core repos share the same structure.
