Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 1 | .. vim: syntax=rst |
| 2 | |
| 3 | Setting Up Aether-in-a-Box |
| 4 | ========================== |
| 5 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 6 | Aether-in-a-Box (AiaB) provides an easy way to deploy Aether's SD-CORE and ROC |
| 7 | components, and then run basic tests to validate the installation. |
| 8 | This guide describes the steps to set up AiaB. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 9 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 10 | AiaB can be set up with a 4G or 5G SD-CORE. In either case, SD-CORE configuration |
| 11 | can be done with or without the ROC. The ROC |
| 12 | provides an interactive GUI for examining and changing the configuration, and is used to |
| 13 | manage the production Aether; it can be deployed to test the integration between |
| 14 | ROC and SD-CORE. If the ROC is not deployed, a simple tool called SimApp |
| 15 | is used to configure the required state in SD-CORE for testing core functionality. |
| 16 | |
| 17 | Helm charts are the primary method of installing the SD-CORE and ROC resources. |
| 18 | AiaB offers a great deal of flexibility regarding which Helm chart versions to install: |
| 19 | |
| 20 | * Local definitions of charts (for testing Helm chart changes) |
| 21 | * Latest published charts (for deploying a development version of Aether) |
| 22 | * Specified versions of charts (for deploying a specific Aether release) |
| 23 | |
| 24 | AiaB can be run on a bare metal machine or VM. System prerequisites: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 25 | |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 26 | * Ubuntu 18.04 clean install |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 27 | * Kernel 4.15 or later |
| 28 | * Haswell CPU or newer |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 29 | * At least 4 CPUs and 12GB RAM |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 30 | * Ability to run "sudo" without a password. Due to this requirement, AiaB is most suited to disposable environments like a VM or a `CloudLab <https://cloudlab.us>`_ machine. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 31 | |
| 32 | Clone Repositories |
| 33 | ------------------ |
| 34 | |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 35 | To initialize the AiaB environment, first clone the following repository:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 36 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 37 | cd ~ |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 38 | git clone "https://gerrit.opencord.org:29418/aether-in-a-box" |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 39 | |
| 40 | If you are going to install AiaB using published Helm charts, you can proceed to the |
| 41 | next section. |
| 42 | |
| 43 | If you wish to install SD-CORE from local Helm charts, clone these additional repositories:: |
| 44 | |
| 45 | mkdir -p ~/cord |
| 46 | cd ~/cord |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 47 | git clone "https://gerrit.opencord.org:29418/sdcore-helm-charts" |
| 48 | git clone "https://gerrit.opencord.org:29418/aether-helm-charts" |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 49 | |
Andy Bavier | 112a33c | 2021-11-11 11:13:49 -0700 | [diff] [blame] | 50 | If you wish to install the ROC from local Helm charts, clone this:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 51 | |
| 52 | mkdir -p ~/cord |
| 53 | cd ~/cord |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 54 | git clone "https://gerrit.opencord.org:29418/roc-helm-charts" |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 55 | |
| 56 | Now change to *~/aether-in-a-box* directory. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 57 | |
| 58 | Set up Authentication Tokens |
| 59 | ---------------------------- |
| 60 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 61 | Edit the file *configs/authentication*. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 62 | |
| 63 | Fill out REGISTRY_USERNAME and REGISTRY_CLI_SECRET as follows: |
| 64 | |
| 65 | * Log into the `Aether Harbor Registry <https://registry.aetherproject.org>`_ using your Crowd credentials |
| 66 | * Select *User Profile* from the drop-down menu in the upper right corner |
| 67 | * For REGISTRY_USERNAME, use the *Username* in your profile |
| 68 | * Copy the *CLI secret* to the clipboard and paste to REGISTRY_CLI_SECRET |
| 69 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 70 | If you have already set up AiaB but you used incorrect credentials, first clean up AiaB as described |
Andy Bavier | 296bfa5 | 2022-02-10 17:48:37 -0700 | [diff] [blame^] | 71 | in the `Cleanup`_ section, and also run:: |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 72 | |
| 73 | kubectl -n omec delete secret aether.registry |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 74 | |
| 75 | Then edit *configs/authentication* and re-build AiaB. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 76 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 77 | Installing the ROC |
| 78 | ------------------ |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 79 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 80 | Note that you must install the ROC *before* installing SD-CORE. |
| 81 | If you are not using the ROC to configure SD-CORE, you can skip this step. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 82 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 83 | First choose whether you will install the 4G or 5G SD-CORE. To install the ROC to |
| 84 | configure the 4G SD-CORE:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 85 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 86 | make roc-4g-models |
| 87 | |
| 88 | To install the ROC to configure the 5G SD-CORE:: |
| 89 | |
| 90 | make roc-5g-models |
| 91 | |
| 92 | By default the above commands install the ROC from the local charts in the Git repos cloned |
| 93 | earlier. In order to install the ROC using the latest published charts, add *CHARTS=latest* |
| 94 | to the command, e.g.,:: |
| 95 | |
| 96 | CHARTS=latest make roc-4g-models |
| 97 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 98 | To install the Aether 1.6 release, add *CHARTS=release-1.6*:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 99 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 100 | CHARTS=release-1.6 make roc-4g-models |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 101 | |
Andy Bavier | 4d27e4e | 2022-02-07 14:49:45 -0700 | [diff] [blame] | 102 | The ROC has successfully initialized when you see output like this:: |
| 103 | |
| 104 | echo "ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm" |
| 105 | ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm |
| 106 | until kubectl -n aether-roc exec pod/onos-cli-5b947f8f6-4r5nm -- \ |
| 107 | curl -s -f -L -X PATCH "http://aether-roc-api:8181/aether-roc-api" \ |
| 108 | --header 'Content-Type: application/json' \ |
| 109 | --data-raw "$(cat /root/aether-in-a-box//roc-5g-models-v4.json)"; do sleep 5; done |
| 110 | command terminated with exit code 22 |
| 111 | command terminated with exit code 22 |
| 112 | command terminated with exit code 22 |
| 113 | "9513ea10-883d-11ec-84bf-721e388172cd" |
| 114 | |
| 115 | Don't worry if you see a few lines of *command terminated with exit code 22*; that command is trying to |
| 116 | load the ROC models, and the message appears if the ROC isn't ready yet. However if you see that message |
| 117 | more than 10 times then something is probably wrong with the ROC or its models. |
| 118 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 119 | Start the 4G SD-CORE |
| 120 | -------------------- |
| 121 | |
| 122 | If you are installing the 5G SD-CORE, you can skip this step. |
| 123 | |
| 124 | To deploy the 4G SD-CORE and run a simple ping test:: |
| 125 | |
| 126 | make test |
| 127 | |
| 128 | By default the above commands install the 4G SD-CORE from the local charts in the Git repos cloned |
| 129 | earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest* |
| 130 | to the command, e.g.,:: |
| 131 | |
| 132 | CHARTS=latest make test |
| 133 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 134 | To install the Aether 1.6 release, add *CHARTS=release-1.6*:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 135 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 136 | CHARTS=release-1.6 make test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 137 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 138 | Start the 5G SD-CORE |
| 139 | -------------------- |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 140 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 141 | If you have already installed the 4G SD-CORE, you must skip this step. Only one version of |
| 142 | the SD-CORE can be installed at a time. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 143 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 144 | To deploy the 5G SD-CORE and run a test with gNBSim that performs Registration + UE-initiated |
| 145 | PDU Session Establishment + sends User Data packets:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 146 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 147 | make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 148 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 149 | By default the above commands install the 5G SD-CORE from the local charts in the Git repos cloned |
| 150 | earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest* |
| 151 | to the command, e.g.,:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 152 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 153 | CHARTS=latest make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 154 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 155 | To install the Aether 1.6 release, add *CHARTS=release-1.6*:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 156 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 157 | CHARTS=release-1.6 make 5g-test |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 158 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 159 | To change the behavior of the test run by gNBSim, change the contents of *gnb.conf* |
| 160 | in *ransim-values.yaml*. Consult the |
| 161 | `gNBSim documentation <https://docs.sd-core.opennetworking.org/master/developer/gnbsim.html>`_ for more information. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 162 | |
Andy Bavier | 4d27e4e | 2022-02-07 14:49:45 -0700 | [diff] [blame] | 163 | Exploring AiaB |
| 164 | -------------- |
| 165 | |
| 166 | The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB. |
| 167 | The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running |
| 168 | in the *aether-roc* namespace. |
| 169 | |
| 170 | The ROC GUI is available on port 31194 on the host running AiaB. |
| 171 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 172 | Cleanup |
| 173 | ------- |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 174 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 175 | The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster. |
| 176 | Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without |
| 177 | destroying the Kubernetes cluster. |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 178 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 179 | * Clean up the 4G SD-CORE: *make reset-test* |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 180 | * Reset the 4G UE / eNB in order to re-run the 4G test: *make reset-ue* |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 181 | * Clean up the 5G SD-CORE: *make reset-5g-test* |
| 182 | * Clean up the ROC: *make roc-clean* |
| 183 | |
| 184 | It's normal for the above commands to take a minute or two to complete. |
| 185 | |
| 186 | As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE |
| 187 | with the ROC. You could run these commands:: |
| 188 | |
| 189 | CHARTS=latest make roc-4g-models # Install ROC with 4G configuration |
| 190 | CHARTS=latest make test # Install 4G SD-CORE and run ping test |
| 191 | make reset-test |
| 192 | make roc-clean |
| 193 | CHARTS=latest make roc-5g-models # Install ROC with 5G configuration |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 194 | CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 195 | make reset-5g-test |
| 196 | make roc-clean |
| 197 | |
| 198 | Developer Loop |
| 199 | -------------- |
| 200 | |
| 201 | Suppose you wish to test a new build of a 5G SD-CORE services. You can deploy custom images |
| 202 | by editing `~/aether-in-a-box/5g-core-values.yaml`, for example:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 203 | |
| 204 | images: |
| 205 | tags: |
| 206 | webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f |
| 207 | pullPolicy: IfNotPresent |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 208 | |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 209 | To upgrade a running 5G SD-CORE with the new image, or to deploy the 5G SD-CORE with the image:: |
Andy Bavier | f73c3d2 | 2021-08-30 10:29:06 -0700 | [diff] [blame] | 210 | |
Andy Bavier | 1d63176 | 2022-01-10 15:47:51 -0800 | [diff] [blame] | 211 | make 5g-test |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 212 | |
| 213 | Troubleshooting / Known Issues |
| 214 | ------------------------------ |
| 215 | |
| 216 | If you suspect a problem, first verify that all pods are in Running state:: |
| 217 | |
| 218 | kubectl -n omec get pods |
| 219 | kubectl -n aether-roc get pods |
| 220 | |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 221 | Pods in ImagePullBackOff State |
| 222 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 223 | If the pods are stuck in ImagePullBackOff state, then it's likely an issue with credentials. To verify this, |
| 224 | run *kubectl describe* on a pod in that state, for example:: |
| 225 | |
| 226 | kubectl -n omec describe pod gnbsim-0 |
| 227 | |
| 228 | Look in the *Events* section for more information about why the image pull failed. If you see *unauthorized to |
| 229 | access repository* then it's probably a credentials issue; see `Set up Authentication Tokens`_ above. |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 230 | |
| 231 | 4G Test Fails |
| 232 | ^^^^^^^^^^^^^ |
| 233 | Occasionally *make test* (for 4G) fails for unknown reasons; this is true regardless of which Helm charts are used. |
Andy Bavier | 7254767 | 2022-02-03 14:45:16 -0700 | [diff] [blame] | 234 | If this happens, first try recreating the simulated UE / eNB and re-running the test as follows:: |
| 235 | |
| 236 | make reset-ue |
| 237 | make test |
| 238 | |
| 239 | If that does not work, try cleaning up AiaB as described above and re-building it. |
| 240 | |
| 241 | If *make test* fails consistently, check whether the configuration has been pushed to the SD-CORE:: |
Andy Bavier | 840e0b4 | 2021-09-16 17:10:23 -0700 | [diff] [blame] | 242 | |
| 243 | kubectl -n omec logs config4g-0 | grep "Successfully" |
| 244 | |
| 245 | You should see that a device group and slice has been pushed:: |
| 246 | |
| 247 | [INFO][WebUI][CONFIG] Successfully posted message for device group 4g-oaisim-user to main config thread |
| 248 | [INFO][WebUI][CONFIG] Successfully posted message for slice default to main config thread |
| 249 | |
| 250 | Then tail the *config4g-0* log and make sure that the configuration has been successfully pushed to all |
| 251 | SD-CORE components. |