Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 1 | .. vim: syntax=rst |
| 2 | |
Andy Bavier | f3e24b6 | 2022-03-30 19:53:01 -0400 | [diff] [blame] | 3 | Aether-in-a-Box for Developers |
| 4 | ============================== |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 5 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 6 | Aether-in-a-Box (AiaB) provides an easy way to deploy Aether's SD-CORE and ROC |
| 7 | components, and then run basic tests to validate the installation. |
| 8 | This guide describes the steps to set up AiaB. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 9 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 10 | AiaB can be set up with a 4G or 5G SD-CORE. In either case, SD-CORE configuration |
| 11 | can be done with or without the ROC. The ROC |
| 12 | provides an interactive GUI for examining and changing the configuration, and is used to |
| 13 | manage the production Aether; it can be deployed to test the integration between |
| 14 | ROC and SD-CORE. If the ROC is not deployed, a simple tool called SimApp |
| 15 | is used to configure the required state in SD-CORE for testing core functionality. |
| 16 | |
| 17 | Helm charts are the primary method of installing the SD-CORE and ROC resources. |
| 18 | AiaB offers a great deal of flexibility regarding which Helm chart versions to install: |
| 19 | |
| 20 | * Local definitions of charts (for testing Helm chart changes) |
| 21 | * Latest published charts (for deploying a development version of Aether) |
| 22 | * Specified versions of charts (for deploying a specific Aether release) |
| 23 | |
| 24 | AiaB can be run on a bare metal machine or VM. System prerequisites: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 25 | |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame] | 26 | * Ubuntu 18.04 clean install |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 27 | * Kernel 4.15 or later |
| 28 | * Haswell CPU or newer |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 29 | * At least 4 CPUs and 12GB RAM |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame] | 30 | * Ability to run "sudo" without a password. Due to this requirement, AiaB is most suited to disposable environments like a VM or a `CloudLab <https://cloudlab.us>`_ machine. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 31 | |
| 32 | Clone Repositories |
| 33 | ------------------ |
| 34 | |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame] | 35 | To initialize the AiaB environment, first clone the following repository:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 36 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 37 | cd ~ |
Hyunsun Moon | 09d63bb | 2022-03-16 09:45:55 -0700 | [diff] [blame] | 38 | git clone "https://gerrit.opencord.org/aether-in-a-box" |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 39 | |
| 40 | If you are going to install AiaB using published Helm charts, you can proceed to the |
| 41 | next section. |
| 42 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 43 | If you wish to install from local Helm charts, clone these additional repositories:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 44 | |
| 45 | mkdir -p ~/cord |
| 46 | cd ~/cord |
Hyunsun Moon | 09d63bb | 2022-03-16 09:45:55 -0700 | [diff] [blame] | 47 | git clone "https://gerrit.opencord.org/sdcore-helm-charts" |
| 48 | git clone "https://gerrit.opencord.org/roc-helm-charts" |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 49 | |
| 50 | Now change to *~/aether-in-a-box* directory. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 51 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 52 | Installing the ROC |
| 53 | ------------------ |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 54 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 55 | Note that you must install the ROC *before* installing SD-CORE. |
| 56 | If you are not using the ROC to configure SD-CORE, you can skip this step. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 57 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 58 | First choose whether you will install the 4G or 5G SD-CORE. To install the ROC to |
| 59 | configure the 4G SD-CORE:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 60 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 61 | make roc-4g-models |
| 62 | |
| 63 | To install the ROC to configure the 5G SD-CORE:: |
| 64 | |
| 65 | make roc-5g-models |
| 66 | |
| 67 | By default the above commands install the ROC from the local charts in the Git repos cloned |
| 68 | earlier. In order to install the ROC using the latest published charts, add *CHARTS=latest* |
| 69 | to the command, e.g.,:: |
| 70 | |
| 71 | CHARTS=latest make roc-4g-models |
| 72 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 73 | To install the Aether 2.0 release, add *CHARTS=release-2.0*:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 74 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 75 | CHARTS=release-2.0 make roc-4g-models |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 76 | |
Andy Bavier | 4d27e4e | 2022-02-07 14:49:45 -0700 | [diff] [blame] | 77 | The ROC has successfully initialized when you see output like this:: |
| 78 | |
| 79 | echo "ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm" |
| 80 | ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm |
| 81 | until kubectl -n aether-roc exec pod/onos-cli-5b947f8f6-4r5nm -- \ |
| 82 | curl -s -f -L -X PATCH "http://aether-roc-api:8181/aether-roc-api" \ |
| 83 | --header 'Content-Type: application/json' \ |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 84 | --data-raw "$(cat /root/aether-in-a-box//roc-5g-models.json)"; do sleep 5; done |
Andy Bavier | 4d27e4e | 2022-02-07 14:49:45 -0700 | [diff] [blame] | 85 | command terminated with exit code 22 |
| 86 | command terminated with exit code 22 |
| 87 | command terminated with exit code 22 |
| 88 | "9513ea10-883d-11ec-84bf-721e388172cd" |
| 89 | |
| 90 | Don't worry if you see a few lines of *command terminated with exit code 22*; that command is trying to |
| 91 | load the ROC models, and the message appears if the ROC isn't ready yet. However if you see that message |
| 92 | more than 10 times then something is probably wrong with the ROC or its models. |
| 93 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 94 | Start the 4G SD-CORE |
| 95 | -------------------- |
| 96 | |
| 97 | If you are installing the 5G SD-CORE, you can skip this step. |
| 98 | |
| 99 | To deploy the 4G SD-CORE and run a simple ping test:: |
| 100 | |
| 101 | make test |
| 102 | |
| 103 | By default the above commands install the 4G SD-CORE from the local charts in the Git repos cloned |
| 104 | earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest* |
| 105 | to the command, e.g.,:: |
| 106 | |
| 107 | CHARTS=latest make test |
| 108 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 109 | To install the Aether 2.0 release, add *CHARTS=release-2.0*:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 110 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 111 | CHARTS=release-2.0 make test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 112 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 113 | Start the 5G SD-CORE |
| 114 | -------------------- |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 115 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 116 | If you have already installed the 4G SD-CORE, you must skip this step. Only one version of |
| 117 | the SD-CORE can be installed at a time. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 118 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 119 | To deploy the 5G SD-CORE and run a test with gNBSim that performs Registration + UE-initiated |
| 120 | PDU Session Establishment + sends User Data packets:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 121 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 122 | make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 123 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 124 | By default the above commands install the 5G SD-CORE from the local charts in the Git repos cloned |
| 125 | earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest* |
| 126 | to the command, e.g.,:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 127 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 128 | CHARTS=latest make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 129 | |
Andy Bavier | 9989213 | 2022-03-11 14:49:10 -0700 | [diff] [blame] | 130 | To install the Aether 2.0 release, add *CHARTS=release-2.0*:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 131 | |
Andy Bavier | 05eca6b | 2022-04-01 15:14:34 -0400 | [diff] [blame^] | 132 | CHARTS=release-2.0 make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 133 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 134 | To change the behavior of the test run by gNBSim, change the contents of *gnb.conf* |
Andy Bavier | 4cda140 | 2022-02-15 15:33:31 -0700 | [diff] [blame] | 135 | in *sd-core-5g-values.yaml*. Consult the |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 136 | `gNBSim documentation <https://docs.sd-core.opennetworking.org/master/developer/gnbsim.html>`_ for more information. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 137 | |
Andy Bavier | 4d27e4e | 2022-02-07 14:49:45 -0700 | [diff] [blame] | 138 | Exploring AiaB |
| 139 | -------------- |
| 140 | |
| 141 | The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB. |
| 142 | The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running |
| 143 | in the *aether-roc* namespace. |
| 144 | |
| 145 | The ROC GUI is available on port 31194 on the host running AiaB. |
| 146 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 147 | Cleanup |
| 148 | ------- |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 149 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 150 | The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster. |
| 151 | Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without |
| 152 | destroying the Kubernetes cluster. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 153 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 154 | * Clean up the 4G SD-CORE: *make reset-test* |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 155 | * Reset the 4G UE / eNB in order to re-run the 4G test: *make reset-ue* |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 156 | * Clean up the 5G SD-CORE: *make reset-5g-test* |
| 157 | * Clean up the ROC: *make roc-clean* |
| 158 | |
| 159 | It's normal for the above commands to take a minute or two to complete. |
| 160 | |
| 161 | As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE |
| 162 | with the ROC. You could run these commands:: |
| 163 | |
| 164 | CHARTS=latest make roc-4g-models # Install ROC with 4G configuration |
| 165 | CHARTS=latest make test # Install 4G SD-CORE and run ping test |
| 166 | make reset-test |
| 167 | make roc-clean |
| 168 | CHARTS=latest make roc-5g-models # Install ROC with 5G configuration |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 169 | CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 170 | make reset-5g-test |
| 171 | make roc-clean |
| 172 | |
| 173 | Developer Loop |
| 174 | -------------- |
| 175 | |
| 176 | Suppose you wish to test a new build of a 5G SD-CORE services. You can deploy custom images |
Andy Bavier | 4cda140 | 2022-02-15 15:33:31 -0700 | [diff] [blame] | 177 | by editing `~/aether-in-a-box/sd-core-5g-values.yaml`, for example:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 178 | |
Andy Bavier | 4cda140 | 2022-02-15 15:33:31 -0700 | [diff] [blame] | 179 | omec-control-plane: |
| 180 | images: |
| 181 | tags: |
| 182 | webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f |
| 183 | pullPolicy: IfNotPresent |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 184 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 185 | To upgrade a running 5G SD-CORE with the new image, or to deploy the 5G SD-CORE with the image:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 186 | |
Andy Bavier | 4cda140 | 2022-02-15 15:33:31 -0700 | [diff] [blame] | 187 | make reset-5g-test; make 5g-test |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 188 | |
| 189 | Troubleshooting / Known Issues |
| 190 | ------------------------------ |
| 191 | |
| 192 | If you suspect a problem, first verify that all pods are in Running state:: |
| 193 | |
| 194 | kubectl -n omec get pods |
| 195 | kubectl -n aether-roc get pods |
| 196 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 197 | 4G Test Fails |
| 198 | ^^^^^^^^^^^^^ |
| 199 | Occasionally *make test* (for 4G) fails for unknown reasons; this is true regardless of which Helm charts are used. |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 200 | If this happens, first try recreating the simulated UE / eNB and re-running the test as follows:: |
| 201 | |
| 202 | make reset-ue |
| 203 | make test |
| 204 | |
| 205 | If that does not work, try cleaning up AiaB as described above and re-building it. |
| 206 | |
| 207 | If *make test* fails consistently, check whether the configuration has been pushed to the SD-CORE:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 208 | |
| 209 | kubectl -n omec logs config4g-0 | grep "Successfully" |
| 210 | |
| 211 | You should see that a device group and slice has been pushed:: |
| 212 | |
| 213 | [INFO][WebUI][CONFIG] Successfully posted message for device group 4g-oaisim-user to main config thread |
| 214 | [INFO][WebUI][CONFIG] Successfully posted message for slice default to main config thread |
| 215 | |
| 216 | Then tail the *config4g-0* log and make sure that the configuration has been successfully pushed to all |
| 217 | SD-CORE components. |