4G/5G document separation
Change-Id: I5270ea6dc695fbade19c1e62be76e628f96c68ee
diff --git a/developer/aiab.rst b/developer/aiab.rst
index a2f278d..633b0d1 100644
--- a/developer/aiab.rst
+++ b/developer/aiab.rst
@@ -32,13 +32,12 @@
* No firewall running on the AiaB host. For example, `sudo ufw status` should show `inactive`,
and `sudo iptables -L` and `sudo nft list` should show a blank configuration.
-**IMPORTANT:**
-
-* Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.
-* AiaB changes the host server by adding systemd-networkd configuration files to the
- host's network configuration. Systemd-networkd is the default networking configuration
- tool for Ubuntu, but if your server or VM uses a different method it may not be fully
- compatible with AiaB.
+.. note::
+ * Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.
+ * AiaB changes the host server by adding systemd-networkd configuration files to the
+ host's network configuration. Systemd-networkd is the default networking configuration
+ tool for Ubuntu, but if your server or VM uses a different method it may not be fully
+ compatible with AiaB.
Ubuntu Environment
------------------
@@ -74,8 +73,11 @@
The latter option can be included in the `.bashrc` file to make it permanent.
-Clone Repositories
-------------------
+Installing the 4G AIAB
+----------------------
+
+Clone 4G AIAB (aether-in-a-box)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To initialize the AiaB environment, first clone the following repository in your home directory::
@@ -83,52 +85,19 @@
git clone "https://gerrit.opencord.org/aether-in-a-box"
cd ~/aether-in-a-box
-**Note**: Most users will install AiaB using *published Helm charts* (e.g., `CHARTS=latest`,
-`CHARTS=release-2.0`) and can proceed to the next section. However, if you need to change the Helm
-charts themselves, clone these additional repositories to work with the *local Helm charts*::
+.. note::
+ * Most users install AiaB using *published Helm charts* (e.g., `CHARTS=latest`, `CHARTS=release-2.0`)
+ * If you wish to modify helm charts and want to test AIAB with modified helm charts then check the
+ section - :ref:`local-helm-4g`
- # Skip this step unless you are developing / changing the Helm charts
- mkdir -p ~/cord
- cd ~/cord
- git clone "https://gerrit.opencord.org/sdcore-helm-charts"
- git clone "https://gerrit.opencord.org/roc-helm-charts"
- cd ~/aether-in-a-box
-.. _rke2-vs-kubespray-install:
-
-RKE2 vs. Kubespray Install
---------------------------
-
-The AiaB installer will bring up Kubernetes on the server where it is run. By default it
-uses `RKE2 <https://docs.rke2.io>`_ as the Kubernetes platform. However, older versions of AiaB
-used `Kubespray <https://kubernetes.io/docs/setup/production-environment/tools/kubespray/>`_
-and that is still an option. To switch to Kubespray as the Kubernetes platform, edit the
-Makefile and replace *rke2* with *kubespray* on this line::
-
- K8S_INSTALL ?= rke2
-
-You may wish to use Kubespray instead of RKE2 if you want to use locally-built images with AiaB
-(e.g., if you are developing SD-CORE services). The reason is that RKE2 uses containerd instead of
-Docker and so cannot access images in the local Docker registry. More details can be found in
-the :ref:`developer-loop` section below.
-
-Installing the ROC
-------------------
+Installing the 4G ROC
+^^^^^^^^^^^^^^^^^^^^^
Note that you must install the ROC *before* installing SD-CORE.
If you are not using the ROC to configure SD-CORE, you can skip this step.
-First choose whether you will install the 4G or 5G SD-CORE. To install the ROC to
-configure the 4G SD-CORE::
-
- make roc-4g-models
-
-To install the ROC to configure the 5G SD-CORE::
-
- make roc-5g-models
-
-By default the above commands install the ROC from the local charts in the Git repos cloned
-earlier. In order to install the ROC using the latest published charts, add *CHARTS=latest*
+In order to install the ROC using the latest published charts, add *CHARTS=latest*
to the command, e.g.,::
CHARTS=latest make roc-4g-models
@@ -137,6 +106,7 @@
CHARTS=release-2.0 make roc-4g-models
+
The ROC has successfully initialized when you see output like this::
echo "ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm"
@@ -154,24 +124,22 @@
load the ROC models, and the message appears if the ROC isn't ready yet. However if you see that message
more than 10 times then something is probably wrong with the ROC or its models.
-Start the 4G SD-CORE
---------------------
-If you are installing the 5G SD-CORE, you can skip this step.
+Installing the 4G SD-CORE
+^^^^^^^^^^^^^^^^^^^^^^^^^
-To deploy the 4G SD-CORE and run a simple ping test::
+If you have already installed the 5G SD-CORE, you must skip this step. Only one version of
+the SD-CORE can be installed at a time.
- make test
-By default the above commands install the 4G SD-CORE from the local charts in the Git repos cloned
-earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest*
+To install the SD-CORE using the latest published charts, add *CHARTS=latest*
to the command, e.g.,::
- CHARTS=latest make test
+ CHARTS=latest make test #override value file - `~/aether-in-a-box/sd-core-4g-values.yaml`
To install the Aether 2.0 release, add *CHARTS=release-2.0*::
- CHARTS=release-2.0 make test
+ CHARTS=release-2.0 make test #override value file - `~/aether-in-a-box/release-2.0/sd-core-4g-values.yaml`
4G SD-CORE deploys the following core components to provide mobile connectivity:
@@ -180,6 +148,7 @@
* PCRF (Policy and Charging Rules Function): Data flow detection, policy enforcement, and flow-based charging.
* MME (Mobility Management Entity): Manages UE access network and mobility, and establishing the bearer path for UE.
* HSS (Home Subscriber Server): The main subscriber database.
+* Config4g (Config Pod)
.. figure:: images/4g-call-flow.png
:align: center
@@ -203,39 +172,271 @@
AiaB deploys a router pod in the "default" namespace with four interfaces: *ran-gw* for the radio network,
*access-gw* for access network, *core-gw* for core network, and *eth0* for the external network.
When a UE starts sending traffics to the data network through the user plane (access network),
-the outgoing data packets traverse the following path across the pods::
+the uplink (UE to internet) data packets traverse the following path across the pods::
(oip1) enb-0 (enb) ==GTP==> (ran-gw) router (access-gw) ==GTP==> (access) upf-0 (core)
----> (core-gw) router (NAT,eth0)
-And the incoming packets follow as::
+And the downlink (internet to UE) packets follow as::
(NAT,eth0) router (core-gw) ----> (core) upf-0 (access) ==GTP==> (access-gw) router (ran-gw)
==GTP==> (enb) enb-0 (oip1)
-**Note:** In the above notations, network interfaces within each pod are shown in parenthesis.
-The IP packets sent/received between the UE and external host via the user plane are GTP-encapsulated
-and tunneled between the eNB and UPF.
+.. note::
+ In the above notations, network interfaces within each pod are shown in parenthesis.
+ The IP packets sent/received between the UE and external host via the user plane are GTP-encapsulated
+ and tunneled between the eNB and UPF.
-For more information, please visit the links below:
+Exploring 4G AIAB
+^^^^^^^^^^^^^^^^^
-* `3GPP TS 36.101 <https://www.etsi.org/deliver/etsi_ts/136100_136199/136101/14.05.00_60/ts_136101v140500p.pdf>`_
-* `4G LTE Concepts and Call Flow <https://www.udemy.com/course/4g-lte-evolved-packet-core-deep-dive-and-call-flows/>`_
+The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB.
+The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running
+in the *aether-roc* namespace.
+
+The ROC GUI is available on port 31194 on the host running AiaB.
+
+See the :ref:`instructions here <developer/aiabhw:Enable Monitoring>` to deploy a basic monitoring stack to AiaB.
+This could be useful if you wish to use AiaB as an environment for prototyping Prometheus exporters or
+Grafana dashboards for Aether.
+
+Cleanup 4G AIAB
+^^^^^^^^^^^^^^^
+
+The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster.
+Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without
+destroying the Kubernetes cluster.
+
+* Clean up the 4G SD-CORE: *make reset-test*
+* Reset the 4G UE / eNB in order to re-run the 4G test: *make reset-ue*
+* Clean up the ROC: *make roc-clean*
+
+It's normal for the above commands to take a minute or two to complete.
+
+As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE
+with the ROC. You could run these commands::
+
+ CHARTS=latest make roc-4g-models # Install ROC with 4G configuration
+ CHARTS=latest make test # Install 4G SD-CORE and run ping test
+ make reset-test
+ make roc-clean
+ CHARTS=latest make roc-5g-models # Install ROC with 5G configuration
+ CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test
+ make reset-5g-test
+ make roc-clean
+
+To completely remove AiaB by tearing down the Kubernetes cluster, run *make clean*.
+
+.. _developer-4g-loop:
+
+Using Custom 4G Container Images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Suppose you wish to test a new build of a 4G SD-CORE services. You can deploy custom images
+by editing::
+
+ Override file - `~/aether-in-a-box/sd-core-4g-values.yaml` if you are using latest or local Helm charts
+ Override file - `~/aether-in-a-box/release-2.0/sd-core-4g-values.yaml` if you are using release-2.0 charts
-Start the 5G SD-CORE
---------------------
+ #update following content in override values to update image tags
+ omec-control-plane:
+ images:
+ repository: "" # default docker hub
+ tags:
+ mme: omecproject/nucleus:master-a8002eb
+ pullPolicy: IfNotPresent
+To upgrade a running 4G SD-CORE with the new image, or to deploy the 4G SD-CORE with the image. Use appropriate
+make commands. Following commands assumes that you are using local helm charts ::
+
+ make reset-test; make test #if you are not using local charts then CHARTS option
+
+**Note**: You can use locally built image (Clone + Compile Code) or you can refer to omecproject
+dockerhub project to see available image tags.
+
+.. _local-helm-4g:
+
+Using Local Helm Charts 4G
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**Note**: Most users will install AiaB using *published Helm charts* (e.g., `CHARTS=latest`,
+`CHARTS=release-2.0`). However, if you need to change the Helm
+charts themselves, clone these additional repositories to work with the *local Helm charts*::
+
+ mkdir -p ~/cord
+ cd ~/cord
+ git clone "https://gerrit.opencord.org/sdcore-helm-charts"
+ git clone "https://gerrit.opencord.org/roc-helm-charts"
+ git clone "https://gerrit.opencord.org/sdfabric-helm-charts"
+ cd ~/aether-in-a-box
+
+Modify the helm charts as per your need. Also execute `helm dep update .` in the changed helm
+chart repo. Example below to add testOpt option in mme.::
+
+ node0:~/cord/sdcore-helm-charts$ git diff
+ diff --git a/omec-control-plane/Chart.yaml b/omec-control-plane/Chart.yaml
+ index 79c3738..48ae901 100644
+ --- a/omec-control-plane/Chart.yaml
+ +++ b/omec-control-plane/Chart.yaml
+ @@ -9,4 +9,4 @@ description: OMEC control plane services
+ name: omec-control-plane
+ icon: https://guide.opencord.org/logos/cord.svg
+
+ -version: 0.11.1
+ +version: 0.11.2
+ diff --git a/omec-control-plane/values.yaml b/omec-control-plane/values.yaml
+ index 33ac6ce..a6b994a 100644
+ --- a/omec-control-plane/values.yaml
+ +++ b/omec-control-plane/values.yaml
+ @@ -395,6 +395,7 @@ config:
+ - id: frequency
+ type: integer
+ mme:
+ + testOpt: true
+ deploy: true
+ podAnnotations:
+ fluentbit.io/parser: mme
+ diff --git a/sdcore-helm-charts/Chart.yaml b/sdcore-helm-charts/Chart.yaml
+ index 44a5558..151eb07 100644
+ --- a/sdcore-helm-charts/Chart.yaml
+ +++ b/sdcore-helm-charts/Chart.yaml
+ @@ -8,7 +8,7 @@ name: sd-core
+ description: SD-Core control plane services
+ icon: https://guide.opencord.org/logos/cord.svg
+ type: application
+ -version: 0.11.8
+ +version: 0.11.9
+ home: https://opennetworking.org/sd-core/
+ maintainers:
+ - name: SD-Core Support
+ @@ -16,9 +16,9 @@ maintainers:
+
+ dependencies:
+ - name: omec-control-plane
+ - version: 0.11.1
+ - repository: https://charts.aetherproject.org
+ - #repository: "file://../omec-control-plane"
+ + version: 0.11.2
+ + #repository: https://charts.aetherproject.org
+ + repository: "file://../omec-control-plane" #refer local helm chart
+ condition: omec-control-plane.enable4G
+
+ - name: omec-sub-provision
+ node0:~/cord/sdcore-helm-charts$
+
+ node0:~$ cd cord/sdcore-helm-charts/omec-control-plane/
+ node0:~/cord/sdcore-helm-charts/omec-control-plane$ helm dependency update .
+
+
+To install the ROC from the local charts::
+
+ make roc-4g-models
+
+To install the 4G SD-CORE from the local charts::
+
+ make test
+
+.. note::
+ * Helm chart changes can not be done when CHARTS option is used. If you need to change helm chart then you should use local helm charts
+
+Troubleshooting 4G Issues
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.**
+
+If you suspect a problem, first verify that all pods are in Running state::
+
+ kubectl -n omec get pods
+ kubectl -n aether-roc get pods
+
+4G Test Fails
+*************
+
+Occasionally *make test* (for 4G) fails for unknown reasons; this is true regardless of which Helm charts are used.
+If this happens, first try recreating the simulated UE / eNB and re-running the test as follows::
+
+ make reset-ue
+ make test
+
+If that does not work, try cleaning up AiaB as described above and re-building it.
+
+If *make test* fails consistently, check whether the configuration has been pushed to the SD-CORE::
+
+ kubectl -n omec logs config4g-0 | grep "Successfully"
+
+You should see that a device group and slice has been pushed::
+
+ [INFO][WebUI][CONFIG] Successfully posted message for device group 4g-oaisim-user to main config thread
+ [INFO][WebUI][CONFIG] Successfully posted message for slice default to main config thread
+
+Then tail the *config4g-0* log and make sure that the configuration has been successfully pushed to all
+SD-CORE components.
+
+
+.. note::
+ For more troubleshooting FAQs, please refer here :ref:`Troubleshooting guide <developer/troubleshooting:Aether-in-a-Box FAQs and Troubleshooting>`
+
+Installing the 5G AIAB
+----------------------
+
+Clone 5G AIAB (aether-in-a-box)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To initialize the AiaB environment, first clone the following repository in your home directory::
+
+ cd ~
+ git clone "https://gerrit.opencord.org/aether-in-a-box"
+ cd ~/aether-in-a-box
+
+.. note::
+ * Most users install AiaB using *published Helm charts* (e.g., `CHARTS=latest`, `CHARTS=release-2.0`)
+ * If you wish to modify helm charts and want to test AIAB with modified helm charts then check the
+ section :ref:`local-helm-5g`
+
+
+Installing the ROC for 5G
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Note that you must install the ROC *before* installing SD-CORE.
+If you are not using the ROC to configure SD-CORE, you can skip this step.
+
+To install the ROC using the latest published charts, add *CHARTS=latest*
+to the command, e.g.,::
+
+ CHARTS=latest make roc-5g-models #override value file - `~/aether-in-a-box/sd-core-5g-values.yaml`
+
+To install the Aether 2.0 release, add *CHARTS=release-2.0*::
+
+ CHARTS=release-2.0 make roc-5g-models #override value file - `~/aether-in-a-box/release-2.0/sd-core-5g-values.yaml`
+
+The ROC has successfully initialized when you see output like this::
+
+ echo "ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm"
+ ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm
+ until kubectl -n aether-roc exec pod/onos-cli-5b947f8f6-4r5nm -- \
+ curl -s -f -L -X PATCH "http://aether-roc-api:8181/aether-roc-api" \
+ --header 'Content-Type: application/json' \
+ --data-raw "$(cat /root/aether-in-a-box//roc-5g-models.json)"; do sleep 5; done
+ command terminated with exit code 22
+ command terminated with exit code 22
+ command terminated with exit code 22
+ "9513ea10-883d-11ec-84bf-721e388172cd"
+
+Don't worry if you see a few lines of *command terminated with exit code 22*; that command is trying to
+load the ROC models, and the message appears if the ROC isn't ready yet. However if you see that message
+more than 10 times then something is probably wrong with the ROC or its models.
+
+
+Installing the 5G SD-CORE
+^^^^^^^^^^^^^^^^^^^^^^^^^
If you have already installed the 4G SD-CORE, you must skip this step. Only one version of
the SD-CORE can be installed at a time.
To deploy the 5G SD-CORE and run a test with gNBSim that performs Registration + UE-initiated
-PDU Session Establishment + sends User Data packets::
+PDU Session Establishment + sends User Data packets.
- make 5g-test
-
-By default the above commands install the 5G SD-CORE from the local charts in the Git repos cloned
-earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest*
+In order to install the SD-CORE using the latest published charts, add *CHARTS=latest*
to the command, e.g.,::
CHARTS=latest make 5g-test
@@ -248,6 +449,200 @@
in *sd-core-5g-values.yaml*. Consult the
`gNBSim documentation <https://docs.sd-core.opennetworking.org/master/developer/gnbsim.html>`_ for more information.
+Exploring 5G AIAB
+^^^^^^^^^^^^^^^^^
+
+The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB.
+The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running
+in the *aether-roc* namespace.
+
+The ROC GUI is available on port 31194 on the host running AiaB.
+
+See the :ref:`instructions here <developer/aiabhw:Enable Monitoring>` to deploy a basic monitoring stack to AiaB.
+This could be useful if you wish to use AiaB as an environment for prototyping Prometheus exporters or
+Grafana dashboards for Aether.
+
+
+Cleanup 5G AIAB
+^^^^^^^^^^^^^^^
+
+The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster.
+Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without
+destroying the Kubernetes cluster.
+
+* Clean up the 5G SD-CORE: *make reset-5g-test*
+* Clean up the ROC: *make roc-clean*
+
+It's normal for the above commands to take a minute or two to complete.
+
+As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE
+with the ROC. You could run these commands::
+
+ CHARTS=latest make roc-4g-models # Install ROC with 4G configuration
+ CHARTS=latest make test # Install 4G SD-CORE and run ping test
+ make reset-test
+ make roc-clean
+ CHARTS=latest make roc-5g-models # Install ROC with 5G configuration
+ CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test
+ make reset-5g-test
+ make roc-clean
+
+To completely remove AiaB by tearing down the Kubernetes cluster, run *make clean*.
+
+.. _developer-5g-loop:
+
+Using Custom 5G Container Images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Suppose you wish to test a new build of a 5G SD-CORE services. You can deploy custom images
+by editing::
+
+
+ Override file - `~/aether-in-a-box/sd-core-5g-values.yaml` if you are using latest or local Helm charts
+ Override file - `~/aether-in-a-box/release-2.0/sd-core-5g-values.yaml` if you are using release-2.0 charts
+
+ #update following content in override values to update image tags
+ 5g-control-plane:
+ images:
+ tags:
+ webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f
+ pullPolicy: IfNotPresent
+
+To upgrade a running 5G SD-CORE with the new image, or to deploy the 5G SD-CORE with the image. Use appropriate
+make commands. Following commands assumes that you are using local helm charts ::
+
+ make reset-5g-test; make 5g-test #if you are not using local charts then use CHARTS option
+
+**Note**: You can use locally built image (Clone + Compile Code) or you can refer to omecproject
+dockerhub project to see available image tags.
+
+.. _local-helm-5g:
+
+Using Local Helm Charts 5G
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**Note**: Most users will install AiaB using *published Helm charts* (e.g., `CHARTS=latest`,
+`CHARTS=release-2.0`). However, if you need to change the Helm
+charts themselves, clone these additional repositories to work with the *local Helm charts*::
+
+ mkdir -p ~/cord
+ cd ~/cord
+ git clone "https://gerrit.opencord.org/sdcore-helm-charts"
+ git clone "https://gerrit.opencord.org/roc-helm-charts"
+ git clone "https://gerrit.opencord.org/sdfabric-helm-charts"
+ cd ~/aether-in-a-box
+
+Modify the helm charts as per your need. Also execute `helm dep update .` in the changed helm
+chart repo. Example below to add testOpt option in amf.::
+
+ node0:~/cord/sdcore-helm-charts$ git diff
+ diff --git a/5g-control-plane/Chart.yaml b/5g-control-plane/Chart.yaml
+ index 421e7e5..3cea334 100644
+ --- a/5g-control-plane/Chart.yaml
+ +++ b/5g-control-plane/Chart.yaml
+ @@ -10,7 +10,7 @@ description: SD-Core 5G control plane services
+ name: 5g-control-plane
+ icon: https://guide.opencord.org/logos/cord.svg
+
+ -version: 0.7.10
+ +version: 0.7.11
+
+ dependencies:
+ - name: mongodb
+ diff --git a/5g-control-plane/values.yaml b/5g-control-plane/values.yaml
+ index 8ddcf66..c15d77d 100644
+ --- a/5g-control-plane/values.yaml
+ +++ b/5g-control-plane/values.yaml
+ @@ -417,6 +417,7 @@ config:
+ ngapIpList:
+ - "0.0.0.0"
+ amf:
+ + testOpt: true
+ deploy: true
+ podAnnotations:
+ field.cattle.io/workloadMetrics: '[{"path":"/metrics","port":9089,"schema":"HTTP"}]'
+ diff --git a/sdcore-helm-charts/Chart.yaml b/sdcore-helm-charts/Chart.yaml
+ index 44a5558..8f52f77 100644
+ --- a/sdcore-helm-charts/Chart.yaml
+ +++ b/sdcore-helm-charts/Chart.yaml
+ @@ -8,7 +8,7 @@ name: sd-core
+ description: SD-Core control plane services
+ icon: https://guide.opencord.org/logos/cord.svg
+ type: application
+ -version: 0.11.8
+ +version: 0.11.9
+ home: https://opennetworking.org/sd-core/
+ maintainers:
+ - name: SD-Core Support
+ @@ -28,9 +28,9 @@ dependencies:
+ condition: omec-sub-provision.enable
+
+ - name: 5g-control-plane
+ - version: 0.7.8
+ - repository: https://charts.aetherproject.org
+ - #repository: "file://../5g-control-plane"
+ + version: 0.7.11
+ + #repository: https://charts.aetherproject.org
+ + repository: "file://../5g-control-plane" #enable this line to refer locally changed helm charts
+ condition: 5g-control-plane.enable5G
+
+ - name: bess-upf
+ node0:~/cord/sdcore-helm-charts$
+
+ node0:~$ cd cord/sdcore-helm-charts/5g-control-plane/
+ node0:~/cord/sdcore-helm-charts/5g-control-plane$ helm dependency update .
+
+To install the ROC from the local charts::
+
+ make roc-5g-models
+
+To install the 5G SD-CORE from the local charts::
+
+ make 5g-test
+
+.. note::
+ * Helm chart changes can not be done when CHARTS option is used. If you need to change helm chart then you should use local helm charts
+
+Troubleshooting 5G Issues
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.**
+
+If you suspect a problem, first verify that all pods are in Running state::
+
+ kubectl -n omec get pods
+ kubectl -n aether-roc get pods
+
+5G Test Fails
+*************
+
+If the 5G test fails (*make 5g-test*) then you will see output like this::
+
+ 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Name: profile2 , Profile Type: pdusessest
+ 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Ue's Passed: 2 , Ue's Failed: 3
+ 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Errors:
+ 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007492, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
+ 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007493, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
+ 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007494, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
+ 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Simulation Result: FAIL
+
+In this case check whether the *webui* pod has restarted... this can happen if it times out waiting
+for the database to come up::
+
+ $ kubectl -n omec get pod -l app=webui
+ NAME READY STATUS RESTARTS AGE
+ webui-6b9c957565-zjqls 1/1 Running 1 (6m55s ago) 7m56s
+
+If the output shows any restarts, then restart the *simapp* pod to cause it to re-push its subscriber state::
+
+ $ kubectl -n omec delete pod -l app=simapp
+ pod "simapp-6c49b87c96-hpf82" deleted
+
+Re-run the 5G test, it should now pass.
+
+.. note::
+ For more troubleshooting FAQs, please refer here :ref:`Troubleshooting guide <developer/troubleshooting:Aether-in-a-Box FAQs and Troubleshooting>`
+
Packet Capture
--------------
@@ -294,164 +689,3 @@
you can explicitly specify the interface along with options (e.g. *-e*). e.g.::
kubectl sniff -n default router -i access-gw -f "-e"
-
-Exploring AiaB
---------------
-
-The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB.
-The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running
-in the *aether-roc* namespace.
-
-The ROC GUI is available on port 31194 on the host running AiaB.
-
-See the :ref:`instructions here <developer/aiabhw:Enable Monitoring>` to deploy a basic monitoring stack to AiaB.
-This could be useful if you wish to use AiaB as an environment for prototyping Prometheus exporters or
-Grafana dashboards for Aether.
-
-Cleanup
--------
-
-The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster.
-Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without
-destroying the Kubernetes cluster.
-
-* Clean up the 4G SD-CORE: *make reset-test*
-* Reset the 4G UE / eNB in order to re-run the 4G test: *make reset-ue*
-* Clean up the 5G SD-CORE: *make reset-5g-test*
-* Clean up the ROC: *make roc-clean*
-
-It's normal for the above commands to take a minute or two to complete.
-
-As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE
-with the ROC. You could run these commands::
-
- CHARTS=latest make roc-4g-models # Install ROC with 4G configuration
- CHARTS=latest make test # Install 4G SD-CORE and run ping test
- make reset-test
- make roc-clean
- CHARTS=latest make roc-5g-models # Install ROC with 5G configuration
- CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test
- make reset-5g-test
- make roc-clean
-
-To completely remove AiaB by tearing down the Kubernetes cluster, run *make clean*.
-
-.. _developer-loop:
-
-Developer Loop
---------------
-
-Suppose you wish to test a new build of a 5G SD-CORE services. You can deploy custom images
-by editing `~/aether-in-a-box/sd-core-5g-values.yaml`, for example::
-
- omec-control-plane:
- images:
- tags:
- webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f
- pullPolicy: IfNotPresent
-
-To upgrade a running 5G SD-CORE with the new image, or to deploy the 5G SD-CORE with the image::
-
- make reset-5g-test; make 5g-test
-
-Note that RKE2 (the default Kubernetes installer) is based on containerd rather than Docker.
-Containerd has its own local image registry that is separate from the local Docker Registry. With RKE2,
-if you have used `docker build` to build a local image, it is only in the Docker registry and so is not
-available to run in AiaB without some additional steps. An easy workaround
-is to use `docker push` to push the image to a remote repository (e.g., Docker Hub) and then modify your
-Helm values file to pull in that remote image. Another option is to save the local Docker image
-into a file and push the file to the containerd registry like this::
-
- docker save -o /tmp/lte-uesoftmodem.tar omecproject/lte-uesoftmodem:1.1.0
- sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io \
- images import /tmp/lte-uesoftmodem.tar
-
-The above commands save the local Docker image `omecproject/lte-uesoftmodem:1.1.0` in a tarball, and then upload
-the tarball into the containerd registry where it is available for use by RKE2. Of course you should replace
-`omecproject/lte-uesoftmodem:1.1.0` with the name of your image.
-
-If you know that you are going to be using AiaB to test locally-built images, probably the easiest thing to do is to
-use the Kubespray installer. If you have already installed using RKE2 and you want to switch to Kubespray, first
-run `make clean` before following the steps in the :ref:`rke2-vs-kubespray-install` section above.
-
-Troubleshooting / Known Issues
-------------------------------
-
-**NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.**
-
-If you suspect a problem, first verify that all pods are in Running state::
-
- kubectl -n omec get pods
- kubectl -n aether-roc get pods
-
-4G Test Fails
-^^^^^^^^^^^^^
-Occasionally *make test* (for 4G) fails for unknown reasons; this is true regardless of which Helm charts are used.
-If this happens, first try recreating the simulated UE / eNB and re-running the test as follows::
-
- make reset-ue
- make test
-
-If that does not work, try cleaning up AiaB as described above and re-building it.
-
-If *make test* fails consistently, check whether the configuration has been pushed to the SD-CORE::
-
- kubectl -n omec logs config4g-0 | grep "Successfully"
-
-You should see that a device group and slice has been pushed::
-
- [INFO][WebUI][CONFIG] Successfully posted message for device group 4g-oaisim-user to main config thread
- [INFO][WebUI][CONFIG] Successfully posted message for slice default to main config thread
-
-Then tail the *config4g-0* log and make sure that the configuration has been successfully pushed to all
-SD-CORE components.
-
-5G Test Fails
-^^^^^^^^^^^^^
-
-If the 5G test fails (*make 5g-test*) then you will see output like this::
-
- 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Name: profile2 , Profile Type: pdusessest
- 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Ue's Passed: 2 , Ue's Failed: 3
- 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Errors:
- 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007492, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
- 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007493, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
- 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007494, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
- 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Simulation Result: FAIL
-
-In this case check whether the *webui* pod has restarted... this can happen if it times out waiting
-for the database to come up::
-
- $ kubectl -n omec get pod -l app=webui
- NAME READY STATUS RESTARTS AGE
- webui-6b9c957565-zjqls 1/1 Running 1 (6m55s ago) 7m56s
-
-If the output shows any restarts, then restart the *simapp* pod to cause it to re-push its subscriber state::
-
- $ kubectl -n omec delete pod -l app=simapp
- pod "simapp-6c49b87c96-hpf82" deleted
-
-Re-run the 5G test, it should now pass.
-
-Proxy Issues
-^^^^^^^^^^^^
-
-When working with AiaB behind a proxy, it may be possible to experience certain issues
-due to security policies. That is, the proxy may block a domain (e.g., opencord.org)
-and you may see messages like these ones when trying to clone or get a copy of aether-in-a-box::
-
- ubuntu18:~$ git clone https://gerrit.opencord.org/aether-in-a-box
- Cloning into 'aether-in-a-box'...
- fatal: unable to access 'https://gerrit.opencord.org/aether-in-a-box/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
-
-or::
-
- ubuntu18:~$ wget https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
- --2022-06-01 13:13:42-- https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
- Resolving proxy.company-xyz.com (proxy.company-xyz.com)... w.x.y.z
- Connecting to proxy.company-xyz.com (proxy.company-xyz.com)|w.x.y.z|:#... connected.
- ERROR: cannot verify gerrit.opencord.org's certificate, issued by 'emailAddress=proxy-team@company-xyz.com,... ,C=US':
- Self-signed certificate encountered.
-
-To address this issue, you need to talk to your company's proxy admins and request to
-unblock (re-classify) the opencord.org domain
diff --git a/developer/aiabhw.rst b/developer/aiabhw.rst
index 6e00d39..7aa142a 100644
--- a/developer/aiabhw.rst
+++ b/developer/aiabhw.rst
@@ -1,7 +1,7 @@
.. vim: syntax=rst
-Aether-in-a-Box on Hardware Radios
-==================================
+Aether-in-a-Box with External 4G Radio
+======================================
This document describes how to set up an Aether-in-a-Box (AiaB) with
a Sercomm eNodeB and connect real devices (e.g., 4G phones). It assumes that
@@ -20,7 +20,7 @@
* Ability to run "sudo" without a password
* No firewall running on the AiaB host
-* 4G or 5G small cell eNodeB
+* 4G small cell eNodeB
* Example: :ref:`Sercomm CBRS LTE small cell eNodeB <edge_deployment/overview:eNB Radio>`
@@ -28,7 +28,7 @@
**IMPORTANT**: AiaB is for simple deployment scenarios and so makes some simplifying assumptions:
-* AiaB assumes either 4G or 5G SD-CORE is deployed. Running both 4G and 5G SD-CORE simultaneously in AiaB
+* AiaB assumes 4G SD-CORE is deployed. Running both 4G and 5G SD-CORE simultaneously in AiaB
is currently not supported. However, running both 4G and 5G SD-CORE simultaneously is supported by Aether.
* Performance and scalability are not goals. AiaB does not support I/O acceleration (e.g., with SR-IOV). However,
performance and scalability are goals of the Aether project.
@@ -83,11 +83,11 @@
IMSIs between 315010999912301 and 315010999912303::
subscribers:
- - ueId-start: 315010999912301
- ueId-end: 315010999912303
- plmnId: 315010
- opc: 69d5c2eb2e2e624750541d3bbc692ba5
- key: 000102030405060708090a0b0c0d0e0f
+ - ueId-start: "315010999912301"
+ ueId-end: "315010999912303"
+ plmnId: "315010"
+ opc: "69d5c2eb2e2e624750541d3bbc692ba5"
+ key: "000102030405060708090a0b0c0d0e0f"
sequenceNumber: 135
Determine which is the interface that has L3 connectivity to the
@@ -104,6 +104,9 @@
Aether. If you don't wish to use the ROC to configure AiaB, you
can skip to the next section.
+.. note::
+ Aether Monitoring is available only when Aether is deployed using ROC.
+
Install AiaB as follows (specifying ``DATA_IFACE`` from above)::
ENABLE_OAISIM=false DATA_IFACE=<iface> CHARTS=latest make roc-4g-models 4g-core
@@ -217,6 +220,8 @@
TX packets 99553 bytes 16646682 (16.6 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
+:: _understanding_aiab_networking
+
Understanding AiaB networking
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -506,90 +511,4 @@
Troubleshooting
---------------
-**NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.**
-
-"make" fails immediately
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-AiaB connects macvlan networks to ``DATA_IFACE`` so that the UPF can communicate on the network.
-To do this it assumes that the *systemd-networkd* service is installed and running, ``DATA_IFACE``
-is under its control, and the systemd-networkd configuration file for ``DATA_IFACE`` ends with
-``<DATA_IFACE>.network``, where ``<DATA_IFACE>`` stands for the actual interface name. It
-tries to find this configuration file by looking in the standard paths. If it fails you'll see
-a message like::
-
- FATAL: Could not find systemd-networkd config for interface foobar, exiting now!
- make: *** [Makefile:112: /users/acb/aether-in-a-box//build/milestones/interface-check] Error 1
-
-In this case, you can specify a ``DATA_IFACE_PATH=<path to the config file>`` argument to ``make``
-so that AiaB can find the systemd-networkd configuration file for ``DATA_IFACE``. It's also possible
-that your system does not use systemd-networkd to configure network interfaces (more likely if you
-are running in a VM), in which case AiaB is currently not able to install in your setup. You
-can check that systemd-networkd is installed and running as follows::
-
- $ systemctl status systemd-networkd.service
- ● systemd-networkd.service - Network Service
- Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
- Active: active (running) since Tue 2022-07-12 13:42:18 CDT; 2h 26min ago
- TriggeredBy: ● systemd-networkd.socket
- Docs: man:systemd-networkd.service(8)
- Main PID: 13777 (systemd-network)
- Status: "Processing requests..."
- Tasks: 1 (limit: 193212)
- Memory: 6.4M
- CGroup: /system.slice/systemd-networkd.service
- └─13777 /lib/systemd/systemd-networkd
-
-Data plane is not working
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The first step is to read `Understanding AiaB networking`_, which gives a high level picture
-of the AiaB data plane and how the pieces fit together. In order to debug the problem you will
-need to figure out where data plane packets from the eNodeB are dropped. One way to do this is to
-run ``tcpdump`` on (1) DATA_IFACE to ensure that the data plane packets are arriving, (2) the
-``access`` interface to see that they make it to the UPF, and (3) the ``core`` to check that they
-are forwarded upstream.
-
-If the upstream packets don't make it to DATA_IFACE, you probably need to add the static route
-on the eNodeB so packets to the UPF have a next hop of DATA_IFACE. You can see these upstream
-packets by running::
-
- tcpdump -i <data-iface> -n udp port 2152
-
-If they don't make it to ``access`` you should check that the kernel routing table is forwarding
-a packet with destination 192.158.252.3 to the ``access`` interface. You can see them by running::
-
- tcpdump -i access -n udp port 2152
-
-If they don't make it to ``core`` then they are being dropped by the UPF for some reason. This
-may be a configuration issue with the state loaded in the ROC / SD-CORE -- the UPF is being told
-to discard these packets. You should check that the device's IMSI is part of a slice and that
-the slice's policy settings allow traffic to that destination. You can view them via the following::
-
- tcpdump -i core -n net 172.250.0.0/16
-
-That command will capture all packets to/from the UE subnet.
-
-If you cannot figure out the issue, see `Getting Help`_.
-
-Restarting the AiaB Server
---------------------------
-
-AiaB should come up in a mostly working state if the AiaB server is rebooted. If any pods are
-stuck in an Error or CrashLoopBackoff state they can be restarted using ``kubectl delete pod``.
-It might also be necessary to power cycle the Sercomm eNodeB in order to get it to reconnect to
-the SD-CORE.
-
-Getting Help
-------------
-
-Please introduce yourself and post your questions to the `#aether-dev` channel on the ONF Community Slack.
-Details about how to join this channel can be found on the `ONF Wiki <https://wiki.opennetworking.org/display/COM/Aether>`_.
-In your introduction please state your institution and position, and describe why you are interested in Aether
-and what is your end goal.
-
-If you need help debugging your setup, please give as much detail as possible about
-your environment: the OS version you have installed, are you running on bare metal or in a VM,
-how much CPU and memory does your server have, are you installing behind a proxy, and so on. Also list the steps
-you have performed so far, and post any error messages you have received. These details will aid the community
-to understand where you are and how to help you make progress.
+Please refer :ref:`Aether Troubleshooting Guide<developer/troubleshooting:Troubleshooting>`
diff --git a/developer/aiabhw5g.rst b/developer/aiabhw5g.rst
new file mode 100644
index 0000000..91dd24c
--- /dev/null
+++ b/developer/aiabhw5g.rst
@@ -0,0 +1,365 @@
+.. vim: syntax=rst
+
+Aether-in-a-Box with External 5G Radio
+=======================================
+
+This document describes how to set up an Aether-in-a-Box (AiaB) with
+a external gNodeB and connect real devices (e.g., 5G phones). It assumes that
+you are already familiar with running AiaB with emulated gNodeB/UE. AiaB on Hardware
+Radios is suitable for laboratory experiments and proof-of-concept deployments.
+Its goals are to provide an easy-to-install environment where Aether's features can be
+explored with real devices. To create this setup you will need the following equipment:
+
+* Server for running AiaB (SD-CORE / UPF / ROC)
+
+ * Ubuntu 18.04 or 20.04 clean install
+ * Haswell CPU family or newer
+ * At least 4 CPUs and 12GB RAM
+ * Internet connection
+ * Ability to run "sudo" without a password
+ * No firewall running on the AiaB host
+
+* External 5G small cell gNodeB or simulator
+
+* If real phones are used then you need SIM card writer and blank SIM cards
+
+**IMPORTANT**: AiaB is for simple deployment scenarios and so makes some simplifying assumptions:
+
+* AiaB assumes either 4G or 5G SD-CORE is deployed. Running both 4G and 5G SD-CORE simultaneously in AiaB
+ is currently not supported. However, running both 4G and 5G SD-CORE simultaneously is supported by Aether.
+* Performance and scalability are not goals. AiaB does not support I/O acceleration (e.g., with SR-IOV). However,
+ performance and scalability are goals of the Aether project.
+* AiaB assumes that the server and the gNodeB are connected to the **same LAN** and
+ share the **same IP subnet**; in other words they can reach each other in a single hop and
+ there is no IP router between them. This simplifies communication between the gNodeB and the UPF,
+ which is running inside a container and has a private IP address that is not necessarily routable
+ on the local network. However, this is not a requirement for all Aether deployments.
+* AiaB also assumes that the AiaB server's network is configured
+ using *systemd-networkd*, which is the default for Ubuntu, and copies some files into `/etc/systemd/network`;
+ the reason for this is to enable persistence of AiaB's networking configuration across server reboots.
+ This configuration method is specific to AiaB.
+
+Preparation
+-----------
+
+Server setup
+------------
+
+The server will run Aether-in-a-Box. The gNodeB will connect to the server over the local network.
+Perform these steps to prepare the server for the AiaB install:
+
+* Connect the server to the local network
+* Perform a clean install of Ubuntu 18.04 or Ubuntu 20.04 on the server
+* Verify that systemd-networkd is being used to configure networking
+ (e.g., run ``systemctl status systemd-networkd.service``)
+* Set up password-less sudo for the user that will install Aether-in-a-Box
+
+After the steps above have been completed, install Aether-in-a-Box as follows::
+
+ sudo apt install git make
+ git clone "https://gerrit.opencord.org/aether-in-a-box"
+ cd aether-in-a-box
+
+Next, modify the file *sd-core-5g-values.yaml*. Under ``subscribers``,
+add an IMSI range for the SIM cards you created, with the Transport Key
+and OPc values you used earlier. For example, the following will add
+IMSIs between 315010999912301 and 315010999912303::
+
+ subscribers:
+ - ueId-start: "315010999912301"
+ ueId-end: "315010999912303"
+ plmnId: "315010"
+ opc: "69d5c2eb2e2e624750541d3bbc692ba5"
+ key: "000102030405060708090a0b0c0d0e0f"
+ sequenceNumber: 135
+
+Determine which is the interface that has L3 connectivity to the
+gNodeB -- this will be ``DATA_IFACE`` in the configuration later. If
+the gNodeB will also be connected to the local network, then this is just the
+server's primary interface. If the gNodeB will be connected via an
+isolated L2/L3 network segment, then ``DATA_IFACE`` refers to the server
+interface on that network. Remember this interface for later.
+
+Option 1: Configure Aether with ROC
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Aether ROC provides a GUI and API for dynamically configuring
+Aether. If you don't wish to use the ROC to configure AiaB, you
+can skip to the next section.
+
+.. note::
+ Aether Monitoring is available only when Aether is deployed using ROC.
+ Monitoring support for 5G is still in progress.
+
+Install AiaB as follows (specifying ``DATA_IFACE`` from above)::
+
+ ENABLE_GNBSIM=false DATA_IFACE=<iface> CHARTS=latest make roc-5g-models 5g-core
+
+Next, use the ROC to add information about your SIM cards.
+The ROC GUI is available at `http://<server-ip>:31194`.
+
+Choose ``Configuration > Site`` from the drop-down at top right and edit
+the ``AiaB site``. Change the following values and click ``Update``:
+
+* MCC: 315
+* MNC: 010
+
+Choose ``Sim Cards`` from the drop-down at top right. Edit the
+existing entries to reflect the SIM cards you are adding to devices
+by replacing their IMSI values. Click ``Update`` after each edit.
+If you want to connect more than two devices, consult the :ref:`ROC
+documentation <operations/subscriber:Configure Connectivity Service for a new Device>`.
+
+Finally, click the Basket icon at top right and click the ``Commit`` button.
+
+Now jump to the `Verifying the AiaB installation`_ section.
+
+
+Option 2: Configure Aether without ROC
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is possible to configure Aether without the ROC,
+using static YAML files and the SimApp service. If you have already
+installed the ROC, you should skip this section.
+
+Edit *sd-core-5g-values.yaml*. Change ``mcc`` and ``mnc`` as follows::
+
+ plmn:
+ mcc: "315"
+ mnc: "010"
+
+Also add the IMSIs of your devices under ``imsis``, for example::
+
+ device-groups:
+ - name: "5g-gnbsim-user"
+ imsis:
+ - "315010999912301"
+ - "315010999912302"
+ - "315010999912303"
+
+Install AiaB as follows (specifying ``DATA_IFACE`` from above)::
+
+ ENABLE_GNBSIM=false DATA_IFACE=<iface> CHARTS=latest make 5g-core
+
+Verifying the AiaB installation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Installing AiaB will take about 20 minutes with a fast Internet
+connection. If you see any errors / timeouts, try running the ``make``
+command again. The build will finish with a message:
+“Your MME IP address is… ” This is just the IP address assigned to
+the ``DATA_IFACE``. Remember this for the gNodeB setup.
+
+When the install is complete, check that the 5G SD-CORE is running
+as follows::
+
+ $ kubectl get pods -n omec
+ NAME READY STATUS RESTARTS AGE
+ amf-6d9d8f44c8-2d7f7 1/1 Running 0 4m6s
+ ausf-9fcbfb6b-b52rm 1/1 Running 0 4m6s
+ mongodb-0 1/1 Running 0 4m6s
+ mongodb-1 1/1 Running 0 3m48s
+ mongodb-arbiter-0 1/1 Running 0 4m6s
+ nrf-5b49c74c7-g7f7f 1/1 Running 0 4m6s
+ nssf-57d6dbc7f8-42ch4 1/1 Running 0 4m6s
+ pcf-dd8b976d4-wwqgm 1/1 Running 0 4m5s
+ simapp-6d7dc8875c-gkvxk 1/1 Running 0 4m6s
+ smf-6476786686-6ptjr 1/1 Running 0 4m6s
+ udm-864ffdf49b-x4gcj 1/1 Running 0 4m6s
+ udr-dc5bf7f5b-xqkvc 1/1 Running 0 4m6s
+ upf-0 5/5 Running 0 4m6s
+ webui-6dc76b5f85-6c65j 1/1 Running 0 4m6s
+ $
+
+You should see all pods in Running status.
+
+If you have installed the ROC, check that all its pods are running
+as follows::
+
+ $ kubectl -n aether-roc get pod
+ NAME READY STATUS RESTARTS AGE
+ aether-roc-api-78cc548bb9-7vjs2 1/1 Running 0 4m16s
+ aether-roc-gui-v2-6d674fd446-tttb5 1/1 Running 0 4m16s
+ aether-roc-umbrella-grafana-74f8489c8f-s9p45 2/2 Running 0 4m16s
+ aether-roc-websocket-855d64549b-44fnc 1/1 Running 0 4m16s
+ onos-cli-5d448ff6c4-stq5t 1/1 Running 0 4m16s
+ onos-config-7f4df96b88-vtp5s 6/6 Running 0 4m16s
+ onos-consensus-store-0 1/1 Running 0 4m15s
+ onos-topo-585c7c8976-6jq7b 3/3 Running 0 4m16s
+ sdcore-adapter-v2-5646d455b9-2d6zl 1/1 Running 0 4m15s
+
+You should see all pods in Running status.
+
+The UPF pod connects to the ``DATA_IFACE`` specified above using macvlan networks called
+``core`` and ``access``. Next, check that these have been successfully created, e.g. using
+``ifconfig``::
+
+ $ ifconfig core
+ core: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
+ inet 192.168.250.1 netmask 255.255.255.0 broadcast 192.168.250.255
+ ether 16:9d:c1:0f:19:3a txqueuelen 1000 (Ethernet)
+ RX packets 513797 bytes 48400525 (48.4 MB)
+ RX errors 0 dropped 0 overruns 0 frame 0
+ TX packets 102996 bytes 26530538 (26.5 MB)
+ TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
+
+ $ ifconfig access
+ access: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
+ inet 192.168.252.1 netmask 255.255.255.0 broadcast 192.168.252.255
+ ether 7a:9f:38:c0:18:15 txqueuelen 1000 (Ethernet)
+ RX packets 558162 bytes 64064410 (64.0 MB)
+ RX errors 0 dropped 0 overruns 0 frame 0
+ TX packets 99553 bytes 16646682 (16.6 MB)
+ TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
+
+Understanding AiaB networking
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Why does AiaB create the ``core`` and ``access`` interfaces? These are necessary to enable
+the UPF to exchange packets with the eNodeB (access) and Internet (core); they correspond to
+the last two network interfaces below inside the UPF's `bessd` container::
+
+ $ kubectl -n omec exec -ti upf-0 bessd -- ip addr
+ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ inet 127.0.0.1/8 scope host lo
+ valid_lft forever preferred_lft forever
+ inet6 ::1/128 scope host
+ valid_lft forever preferred_lft forever
+ 3: eth0@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
+ link/ether 8a:e2:64:10:4e:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
+ inet 192.168.84.19/32 scope global eth0
+ valid_lft forever preferred_lft forever
+ inet6 fe80::88e2:64ff:fe10:4ebe/64 scope link
+ valid_lft forever preferred_lft forever
+ 4: access@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
+ link/ether 82:b4:ea:00:50:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
+ inet 192.168.252.3/24 brd 192.168.252.255 scope global access
+ valid_lft forever preferred_lft forever
+ inet6 fe80::80b4:eaff:fe00:503e/64 scope link
+ valid_lft forever preferred_lft forever
+ 5: core@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
+ link/ether 4e:ac:69:31:a3:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0
+ inet 192.168.250.3/24 brd 192.168.250.255 scope global core
+ valid_lft forever preferred_lft forever
+ inet6 fe80::4cac:69ff:fe31:a388/64 scope link
+ valid_lft forever preferred_lft forever
+
+In other words, there are interfaces named ``access`` and ``core`` **both inside and outside** the UPF. All four
+are MACVLAN interfaces
+bridged with DATA_IFACE. There are two subnets on this bridge: the two ``access`` interfaces are on 192.168.252.0/24
+and the two ``core`` interfaces are on 192.168.250.0/24. It is helpful to think of two links, called
+``access`` and ``core``, connecting the AiaB host and UPF. AiaB sets up IP routes on the AiaB host and inside the UPF
+to forward packets into and out of the UPF as explained below.
+
+The ``access`` interface **inside the UPF** has an IP address of 192.168.252.3; this is the destination IP address
+of GTP-encapsulated data plane packets from the gNodeB. In order for these packets to actually find their way
+to the UPF, they must arrive on the DATA_IFACE interface and then be forwarded on the ``access`` interface
+**outside the UPF**.
+The next section describes how to configure a static route on the gNodeB in order to send the GTP packets to
+DATA_IFACE. Forwarding the packets to the ``access`` interface is done by the following kernel route on the
+AiaB host (which should be present if your AiaB installation was successful)::
+
+ $ route -n | grep "Iface\|access"
+ Destination Gateway Genmask Flags Metric Ref Use Iface
+ 192.168.252.0 0.0.0.0 255.255.255.0 U 0 0 0 access
+
+The high-level behavior of the UPF is to forward packets between its ``access`` to ``core`` interfaces, while
+at the same time removing/adding GTP encapsulation on the ``access`` side. Upstream packets
+arriving on the ``access`` side from a UE have their GTP headers removed and the raw IP packets are
+forwarded to the ``core`` interface. The routes inside the UPF's `bessd` container will look something
+like this::
+
+ $ kubectl -n omec exec -ti upf-0 -c bessd -- ip route
+ default via 169.254.1.1 dev eth0
+ default via 192.168.250.1 dev core metric 110
+ 128.105.144.0/22 via 192.168.252.1 dev access
+ 128.105.145.141 via 169.254.1.1 dev eth0
+ 169.254.1.1 dev eth0 scope link
+ 192.168.250.0/24 dev core proto kernel scope link src 192.168.250.3
+ 192.168.252.0/24 dev access proto kernel scope link src 192.168.252.3
+
+The default route via 192.168.250.1 is directing upstream packets to the Internet via the ``core`` interface,
+with a next hop of the ``core`` interface **outside the UPF**.
+These packets undergo source NAT in the kernel (also configured by AiaB) and are sent to the IP destination
+in the packet. The return (downstream) packets undergo reverse NAT and now have a destination IP address of the UE.
+They are forwarded by the kernel to the ``core`` interface by these rules on the AiaB host::
+
+ $ route -n | grep "Iface\|core"
+ Destination Gateway Genmask Flags Metric Ref Use Iface
+ 172.250.0.0 192.168.250.3 255.255.0.0 UG 0 0 0 core
+ 192.168.250.0 0.0.0.0 255.255.255.0 U 0 0 0 core
+
+The first rule above matches packets to the UEs (on 172.250.0.0/16 subnet). The next hop for these
+packets is the ``core`` IP address **inside the UPF**. The second rule says that next hop address is
+reachable on the ``core`` interface **outside the UPF**. As a result the downstream packets arrive in the
+UPF where they
+are GTP-encapsulated with the IP address of the gNodeB. Inside the UPF these packets will match a route
+like this one (see above; 128.105.144.0/22 in this case is the DATA_IFACE subnet)::
+
+ 128.105.144.0/22 via 192.168.252.1 dev access
+
+These packets are forwarded to the ``access`` interface **outside the UPF** and out DATA_IFACE to the eNodeB.
+Recall that AiaB assumes that the eNodeB is on the same subnet as DATA_IFACE, so in this case it also has an
+IP address in the 128.105.144.0/22 range.
+
+gNodeB setup
+------------
+
+We expect external gNodeB configuration is carried out separately.
+
+Connect the gNodeB LAN port to a free Ethernet port on a Linux machine
+(say, a laptop) that will be used for the initial configuration of
+the gNodeB.
+
+- Test connectivity from gNodeB to the UPF. If required add a static route to the UPF address (192.168.252.3)
+- Test connectivity from the gNodeB to the AMF
+
+If connectivity results are success, then you are ready to try to connect devices to the network.
+
+Connecting Devices
+------------------
+
+Documenting how to configure different types of devices to work
+with Aether is work-in-progress, but here are some basic guidelines.
+
+Create SIM cards by following the instructions for your SIM card writer.
+Of course you are free to use any values for IMSI, etc. that you choose,
+but these are the values that will work with the rest of the configuration
+in this document:
+
+* IMSI: each one is unique, matching pattern ``315010*********`` (15 digits)
+* OPc: ``69d5c2eb2e2e624750541d3bbc692ba5``
+* Transport Key: ``000102030405060708090a0b0c0d0e0f``
+
+If you choose different values for your SIM cards, you will need to
+modify subsequent configuration steps appropriately.
+
+Insert the SIM cards in devices that you wish to be able to connect to the Aether network.
+
+
+The values of IMSI, OPc, and Transport Key you have configured on your SIM cards
+must be entered into the ``subscribers`` block under ``omec-sub-provision`` in the
+``sd-core-4g-values.yaml`` file. If you are not using the ROC, the IMSIs must also be
+added under ``device-groups``, and the relevant device group added under ``network-slices``.
+If you are using the ROC, then your devices must be configured there and the associated
+device group added to a slice. In either case it is necessary to configure the basic info
+under ``subscribers``.
+
+Be aware that not all phones support the CBRS frequency bands. AiaB is known to work
+with recent iPhones (11 and greater) and Google Pixel phones (4 and up). CBRS may also be
+supported by recent phones from Samsung, LG Electronics and Motorola Mobility, but these have
+not been tested with AiaB. If you successfully test a phone on AiaB, please post details on
+Slack so we can add it to the list.
+
+The APN to configure on your phone is ``internet``.
+
+Enable Monitoring
+-----------------
+
+Support for monitoring dashboard is work in progress for 5G
+
+Troubleshooting
+---------------
+
+Please refer :ref:`Aether Troubleshooting Guide<developer/troubleshooting:Troubleshooting>`
diff --git a/developer/troubleshooting.rst b/developer/troubleshooting.rst
new file mode 100644
index 0000000..af7e1b3
--- /dev/null
+++ b/developer/troubleshooting.rst
@@ -0,0 +1,238 @@
+.. vim: syntax=rst
+
+.. _aiab_troubleshooting:
+
+Aether-in-a-Box FAQs and Troubleshooting
+========================================
+
+FAQs
+----
+
+RKE2 vs. Kubespray Install
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The AiaB installer will bring up Kubernetes on the server where it is run. By default it
+uses `RKE2 <https://docs.rke2.io>`_ as the Kubernetes platform. However, older versions of AiaB
+used `Kubespray <https://kubernetes.io/docs/setup/production-environment/tools/kubespray/>`_
+and that is still an option. To switch to Kubespray as the Kubernetes platform, edit the
+Makefile and replace *rke2* with *kubespray* on this line::
+
+ node0:~/aether-in-a-box$ git diff Makefile
+ diff --git a/Makefile b/Makefile
+ index 5f2c186..608c221 100644
+ --- a/Makefile
+ +++ b/Makefile
+ @@ -35,7 +35,7 @@ ENABLE_GNBSIM ?= true
+ ENABLE_SUBSCRIBER_PROXY ?= false
+ GNBSIM_COLORS ?= true
+
+ -K8S_INSTALL ?= rke2
+ +K8S_INSTALL ?= kubespray
+ CTR_CMD := sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io
+
+ PROXY_ENABLED ?= false
+ node0:~/aether-in-a-box$
+
+
+You may wish to use Kubespray instead of RKE2 if you want to use locally-built images with AiaB
+(e.g., if you are developing SD-CORE services). The reason is that RKE2 uses containerd instead of
+Docker and so cannot access images in the local Docker registry.
+
+How to use Local Image
+^^^^^^^^^^^^^^^^^^^^^^
+
+Note that RKE2 (the default Kubernetes installer) is based on containerd rather than Docker.
+Containerd has its own local image registry that is separate from the local Docker Registry. With RKE2,
+if you have used `docker build` to build a local image, it is only in the Docker registry and so is not
+available to run in AiaB without some additional steps. An easy workaround
+is to use `docker push` to push the image to a remote repository (e.g., Docker Hub) and then modify your
+Helm values file to pull in that remote image. Another option is to save the local Docker image
+into a file and push the file to the containerd registry like this::
+
+ docker save -o /tmp/lte-uesoftmodem.tar omecproject/lte-uesoftmodem:1.1.0
+ sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io \
+ images import /tmp/lte-uesoftmodem.tar
+
+The above commands save the local Docker image `omecproject/lte-uesoftmodem:1.1.0` in a tarball, and then upload
+the tarball into the containerd registry where it is available for use by RKE2. Of course you should replace
+`omecproject/lte-uesoftmodem:1.1.0` with the name of your image.
+
+If you know that you are going to be using AiaB to test locally-built images, probably the easiest thing to do is to
+use the Kubespray installer. If you have already installed using RKE2 and you want to switch to Kubespray, first
+run `make clean` before following the steps in the :ref:`rke2-vs-kubespray-install` section above.
+
+Restarting the AiaB Server
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+AiaB should come up in a mostly working state if the AiaB server is rebooted. If any pods are
+stuck in an Error or CrashLoopBackoff state they can be restarted using ``kubectl delete pod``.
+It might also be necessary to power cycle the Sercomm eNodeB in order to get it to reconnect to
+the SD-CORE.
+
+
+Enabling externalIP at MME
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can enable externalIP service in the MME by providing following config in the override file::
+
+ node0:~/aether-in-a-box$ git diff sd-core-4g-values.yaml
+ diff --git a/sd-core-4g-values.yaml b/sd-core-4g-values.yaml
+ index 0939739..f240f89 100644
+ --- a/sd-core-4g-values.yaml
+ +++ b/sd-core-4g-values.yaml
+ @@ -24,6 +24,11 @@ omec-control-plane:
+ bootstrap:
+ users: []
+ staticusers: []
+ + mme:
+ + s1ap:
+ + serviceType: ClusterIP
+ + externalIP: 10.1.1.1
+ +
+ spgwc:
+ pfcp: true
+ cfgFiles:
+ node0:~/aether-in-a-box$
+
+Enabling externalIP at AMF
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can enable externalIP service in the AMF by providing following config in the override file::
+
+ node0:~/aether-in-a-box$ git diff sd-core-5g-values.yaml
+ diff --git a/sd-core-5g-values.yaml b/sd-core-5g-values.yaml
+ index e513e1f..fc1c684 100644
+ --- a/sd-core-5g-values.yaml
+ +++ b/sd-core-5g-values.yaml
+ @@ -34,6 +34,9 @@
+
+ amf:
+ cfgFiles:
+ + ngapp:
+ + serviceType: ClusterIP
+ + externalIp: "10.1.1.2"
+ + port: 38412
+ amfcfg.conf:
+ configuration:
+ enableDBStore: false
+ @@ -176,6 +179,7 @@ omec-user-plane:
+ cpiface:
+ dnn: "internet"
+ hostname: "upf"
+
+ 5g-ran-sim:
+ enable: ${ENABLE_GNBSIM}
+
+ node0:~/aether-in-a-box$
+
+Troubleshooting
+---------------
+
+**NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.**
+
+Proxy Issues
+^^^^^^^^^^^^
+
+When working with AiaB behind a proxy, it may be possible to experience certain issues
+due to security policies. That is, the proxy may block a domain (e.g., opencord.org)
+and you may see messages like these ones when trying to clone or get a copy of aether-in-a-box::
+
+ ubuntu18:~$ git clone https://gerrit.opencord.org/aether-in-a-box
+ Cloning into 'aether-in-a-box'...
+ fatal: unable to access 'https://gerrit.opencord.org/aether-in-a-box/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
+
+or::
+
+ ubuntu18:~$ wget https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
+ --2022-06-01 13:13:42-- https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
+ Resolving proxy.company-xyz.com (proxy.company-xyz.com)... w.x.y.z
+ Connecting to proxy.company-xyz.com (proxy.company-xyz.com)|w.x.y.z|:#... connected.
+ ERROR: cannot verify gerrit.opencord.org's certificate, issued by 'emailAddress=proxy-team@company-xyz.com,... ,C=US':
+ Self-signed certificate encountered.
+
+To address this issue, you need to talk to your company's proxy admins and request to
+unblock (re-classify) the opencord.org domain
+
+
+"make" fails immediately
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+AiaB connects macvlan networks to ``DATA_IFACE`` so that the UPF can communicate on the network.
+To do this it assumes that the *systemd-networkd* service is installed and running, ``DATA_IFACE``
+is under its control, and the systemd-networkd configuration file for ``DATA_IFACE`` ends with
+``<DATA_IFACE>.network``, where ``<DATA_IFACE>`` stands for the actual interface name. It
+tries to find this configuration file by looking in the standard paths. If it fails you'll see
+a message like::
+
+ FATAL: Could not find systemd-networkd config for interface foobar, exiting now!
+ make: *** [Makefile:112: /users/acb/aether-in-a-box//build/milestones/interface-check] Error 1
+
+In this case, you can specify a ``DATA_IFACE_PATH=<path to the config file>`` argument to ``make``
+so that AiaB can find the systemd-networkd configuration file for ``DATA_IFACE``. It's also possible
+that your system does not use systemd-networkd to configure network interfaces (more likely if you
+are running in a VM), in which case AiaB is currently not able to install in your setup. You
+can check that systemd-networkd is installed and running as follows::
+
+ $ systemctl status systemd-networkd.service
+ ● systemd-networkd.service - Network Service
+ Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
+ Active: active (running) since Tue 2022-07-12 13:42:18 CDT; 2h 26min ago
+ TriggeredBy: ● systemd-networkd.socket
+ Docs: man:systemd-networkd.service(8)
+ Main PID: 13777 (systemd-network)
+ Status: "Processing requests..."
+ Tasks: 1 (limit: 193212)
+ Memory: 6.4M
+ CGroup: /system.slice/systemd-networkd.service
+ └─13777 /lib/systemd/systemd-networkd
+
+Data plane is not working
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The first step is to read `Understanding AiaB networking`_understanding_aiab_networking, which
+gives a high level picture
+of the AiaB data plane and how the pieces fit together. In order to debug the problem you will
+need to figure out where data plane packets from the eNodeB are dropped. One way to do this is to
+run ``tcpdump`` on (1) DATA_IFACE to ensure that the data plane packets are arriving, (2) the
+``access`` interface to see that they make it to the UPF, and (3) the ``core`` to check that they
+are forwarded upstream.
+
+If the upstream packets don't make it to DATA_IFACE, you probably need to add the static route
+on the eNodeB so packets to the UPF have a next hop of DATA_IFACE. You can see these upstream
+packets by running::
+
+ tcpdump -i <data-iface> -n udp port 2152
+
+If they don't make it to ``access`` you should check that the kernel routing table is forwarding
+a packet with destination 192.158.252.3 to the ``access`` interface. You can see them by running::
+
+ tcpdump -i access -n udp port 2152
+
+If they don't make it to ``core`` then they are being dropped by the UPF for some reason. This
+may be a configuration issue with the state loaded in the ROC / SD-CORE -- the UPF is being told
+to discard these packets. You should check that the device's IMSI is part of a slice and that
+the slice's policy settings allow traffic to that destination. You can view them via the following::
+
+ tcpdump -i core -n net 172.250.0.0/16
+
+That command will capture all packets to/from the UE subnet.
+
+If you cannot figure out the issue, see `Getting Help`_.
+
+.. _rke2-vs-kubespray-install:
+
+Getting Help
+------------
+
+Please introduce yourself and post your questions to the `#aether-dev` channel on the ONF Community Slack.
+Details about how to join this channel can be found on the `ONF Wiki <https://wiki.opennetworking.org/display/COM/Aether>`_.
+In your introduction please state your institution and position, and describe why you are interested in Aether
+and what is your end goal.
+
+If you need help debugging your setup, please give as much detail as possible about
+your environment: the OS version you have installed, are you running on bare metal or in a VM,
+how much CPU and memory does your server have, are you installing behind a proxy, and so on. Also list the steps
+you have performed so far, and post any error messages you have received. These details will aid the community
+to understand where you are and how to help you make progress.
+
+
diff --git a/dict.txt b/dict.txt
index 70e049e..c13f0d6 100644
--- a/dict.txt
+++ b/dict.txt
@@ -80,12 +80,14 @@
Wireshark
YAML
aether
+aiab
alicea
allocatable
amf
analytics
ansible
api
+aren't
arp
atomix
ausf
@@ -125,6 +127,7 @@
differentiator
disaggregated
dl
+dockerhub
doesn
downlink
dropdown
@@ -137,6 +140,7 @@
enb
etcd
ethernet
+externalIP
fredf
func
gNB
@@ -161,6 +165,7 @@
imei
incrementing
ip
+isn't
jitter
jjb
json
@@ -202,6 +207,7 @@
nrf
nssf
omec
+omecproject
onboarding
onlab
onos
@@ -250,6 +256,7 @@
sdcore
sdfabric
seattle
+shouldn't
simapp
smf
smgmt
@@ -267,6 +274,7 @@
tcp
tcpdump
telegraf
+testOpt
tfvars
topo
topologies
diff --git a/edge_deployment/fabric_switch_bootstrap.rst b/edge_deployment/fabric_switch_bootstrap.rst
index 0e6b114..833f0f5 100644
--- a/edge_deployment/fabric_switch_bootstrap.rst
+++ b/edge_deployment/fabric_switch_bootstrap.rst
@@ -21,7 +21,7 @@
Installing Open Network Linux
-----------------------------
-See :ref:`Provision Switches <sdfabric:deployment:step 1: provision switches>`
+See :ref:`Provision Switches <sdfabric:deployment_guide>`
to learn about how to enter ONIE Rescue mode and install Open Network Linux on the switches.
Please return here and continue the rest of the step once you finish ONL installation.
diff --git a/index.rst b/index.rst
index e5229fa..af7b227 100644
--- a/index.rst
+++ b/index.rst
@@ -13,7 +13,9 @@
* Setup an Aether software development environment with :doc:`Aether-in-a-Box </developer/aiab>`.
-* For a PoC deployment, bring up an :doc:`Aether-in-a-Box on Real Radios <developer/aiabhw>`.
+* For a PoC deployment, bring up an :doc:`Aether-in-a-Box on 4G Real Radios <developer/aiabhw>`.
+
+* For a PoC deployment, bring up an :doc:`Aether-in-a-Box on 5G Real Radios <developer/aiabhw5g>`.
* Learn about how to :doc:`configure Aether using the ROC </operations/gui>`.
@@ -61,6 +63,8 @@
developer/aiab
developer/aiabhw
+ developer/aiabhw5g
+ developer/troubleshooting
developer/contributing.rst
.. toctree::