blob: 12defd3db6ab61460f4971cb833d58e20ebab997 [file] [log] [blame]
Andy Bavierf73c3d22021-08-30 10:29:06 -07001.. vim: syntax=rst
2
Andy Bavierf3e24b62022-03-30 19:53:01 -04003Aether-in-a-Box for Developers
4==============================
Andy Bavierf73c3d22021-08-30 10:29:06 -07005
Andy Bavier840e0b42021-09-16 17:10:23 -07006Aether-in-a-Box (AiaB) provides an easy way to deploy Aether's SD-CORE and ROC
7components, and then run basic tests to validate the installation.
8This guide describes the steps to set up AiaB.
Andy Bavierf73c3d22021-08-30 10:29:06 -07009
Andy Bavier840e0b42021-09-16 17:10:23 -070010AiaB can be set up with a 4G or 5G SD-CORE. In either case, SD-CORE configuration
11can be done with or without the ROC. The ROC
12provides an interactive GUI for examining and changing the configuration, and is used to
13manage the production Aether; it can be deployed to test the integration between
14ROC and SD-CORE. If the ROC is not deployed, a simple tool called SimApp
15is used to configure the required state in SD-CORE for testing core functionality.
16
17Helm charts are the primary method of installing the SD-CORE and ROC resources.
18AiaB offers a great deal of flexibility regarding which Helm chart versions to install:
19
20* Local definitions of charts (for testing Helm chart changes)
21* Latest published charts (for deploying a development version of Aether)
22* Specified versions of charts (for deploying a specific Aether release)
23
24AiaB can be run on a bare metal machine or VM. System prerequisites:
Andy Bavierf73c3d22021-08-30 10:29:06 -070025
Andy Bavier296bfa52022-02-10 17:48:37 -070026* Ubuntu 18.04 clean install
Andy Bavierf73c3d22021-08-30 10:29:06 -070027* Kernel 4.15 or later
28* Haswell CPU or newer
Andy Bavier72547672022-02-03 14:45:16 -070029* At least 4 CPUs and 12GB RAM
Andy Bavier296bfa52022-02-10 17:48:37 -070030* Ability to run "sudo" without a password. Due to this requirement, AiaB is most suited to disposable environments like a VM or a `CloudLab <https://cloudlab.us>`_ machine.
Andy Bavierf73c3d22021-08-30 10:29:06 -070031
32Clone Repositories
33------------------
34
Andy Bavier296bfa52022-02-10 17:48:37 -070035To initialize the AiaB environment, first clone the following repository::
Andy Bavierf73c3d22021-08-30 10:29:06 -070036
Andy Bavier840e0b42021-09-16 17:10:23 -070037 cd ~
Hyunsun Moon09d63bb2022-03-16 09:45:55 -070038 git clone "https://gerrit.opencord.org/aether-in-a-box"
Andy Bavier840e0b42021-09-16 17:10:23 -070039
40If you are going to install AiaB using published Helm charts, you can proceed to the
41next section.
42
Andy Bavier99892132022-03-11 14:49:10 -070043If you wish to install from local Helm charts, clone these additional repositories::
Andy Bavier840e0b42021-09-16 17:10:23 -070044
45 mkdir -p ~/cord
46 cd ~/cord
Hyunsun Moon09d63bb2022-03-16 09:45:55 -070047 git clone "https://gerrit.opencord.org/sdcore-helm-charts"
48 git clone "https://gerrit.opencord.org/roc-helm-charts"
Andy Bavier840e0b42021-09-16 17:10:23 -070049
50Now change to *~/aether-in-a-box* directory.
Andy Bavierf73c3d22021-08-30 10:29:06 -070051
Andy Bavier62678402022-04-18 12:06:52 -070052RKE2 vs. Kubespray Install
53--------------------------
54
55The AiaB installer will bring up Kubernetes on the server where it is run. By default it
56uses `RKE2 <https://docs.rke2.io>`_ as the Kubernetes platform. However, older versions of AiaB
57used `Kubespray <https://kubernetes.io/docs/setup/production-environment/tools/kubespray/>`_
58and that is still an option. To switch to Kubespray as the Kubernetes platform, edit the
59Makefile and replace *rke2* with *kubespray* on this line::
60
61 K8S_INSTALL := rke2
62
Andy Bavier840e0b42021-09-16 17:10:23 -070063Installing the ROC
64------------------
Andy Bavierf73c3d22021-08-30 10:29:06 -070065
Andy Bavier840e0b42021-09-16 17:10:23 -070066Note that you must install the ROC *before* installing SD-CORE.
67If you are not using the ROC to configure SD-CORE, you can skip this step.
Andy Bavierf73c3d22021-08-30 10:29:06 -070068
Andy Bavier840e0b42021-09-16 17:10:23 -070069First choose whether you will install the 4G or 5G SD-CORE. To install the ROC to
70configure the 4G SD-CORE::
Andy Bavierf73c3d22021-08-30 10:29:06 -070071
Andy Bavier840e0b42021-09-16 17:10:23 -070072 make roc-4g-models
73
74To install the ROC to configure the 5G SD-CORE::
75
76 make roc-5g-models
77
78By default the above commands install the ROC from the local charts in the Git repos cloned
79earlier. In order to install the ROC using the latest published charts, add *CHARTS=latest*
80to the command, e.g.,::
81
82 CHARTS=latest make roc-4g-models
83
Andy Bavier99892132022-03-11 14:49:10 -070084To install the Aether 2.0 release, add *CHARTS=release-2.0*::
Andy Bavier840e0b42021-09-16 17:10:23 -070085
Andy Bavier99892132022-03-11 14:49:10 -070086 CHARTS=release-2.0 make roc-4g-models
Andy Bavier840e0b42021-09-16 17:10:23 -070087
Andy Bavier4d27e4e2022-02-07 14:49:45 -070088The ROC has successfully initialized when you see output like this::
89
90 echo "ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm"
91 ONOS CLI pod: pod/onos-cli-5b947f8f6-4r5nm
92 until kubectl -n aether-roc exec pod/onos-cli-5b947f8f6-4r5nm -- \
93 curl -s -f -L -X PATCH "http://aether-roc-api:8181/aether-roc-api" \
94 --header 'Content-Type: application/json' \
Andy Bavier99892132022-03-11 14:49:10 -070095 --data-raw "$(cat /root/aether-in-a-box//roc-5g-models.json)"; do sleep 5; done
Andy Bavier4d27e4e2022-02-07 14:49:45 -070096 command terminated with exit code 22
97 command terminated with exit code 22
98 command terminated with exit code 22
99 "9513ea10-883d-11ec-84bf-721e388172cd"
100
101Don't worry if you see a few lines of *command terminated with exit code 22*; that command is trying to
102load the ROC models, and the message appears if the ROC isn't ready yet. However if you see that message
103more than 10 times then something is probably wrong with the ROC or its models.
104
Andy Bavier840e0b42021-09-16 17:10:23 -0700105Start the 4G SD-CORE
106--------------------
107
108If you are installing the 5G SD-CORE, you can skip this step.
109
110To deploy the 4G SD-CORE and run a simple ping test::
111
112 make test
113
114By default the above commands install the 4G SD-CORE from the local charts in the Git repos cloned
115earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest*
116to the command, e.g.,::
117
118 CHARTS=latest make test
119
Andy Bavier99892132022-03-11 14:49:10 -0700120To install the Aether 2.0 release, add *CHARTS=release-2.0*::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700121
Andy Bavier99892132022-03-11 14:49:10 -0700122 CHARTS=release-2.0 make test
Andy Bavierf73c3d22021-08-30 10:29:06 -0700123
Andy Bavier840e0b42021-09-16 17:10:23 -0700124Start the 5G SD-CORE
125--------------------
Andy Bavierf73c3d22021-08-30 10:29:06 -0700126
Andy Bavier840e0b42021-09-16 17:10:23 -0700127If you have already installed the 4G SD-CORE, you must skip this step. Only one version of
128the SD-CORE can be installed at a time.
Andy Bavierf73c3d22021-08-30 10:29:06 -0700129
Andy Bavier1d631762022-01-10 15:47:51 -0800130To deploy the 5G SD-CORE and run a test with gNBSim that performs Registration + UE-initiated
131PDU Session Establishment + sends User Data packets::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700132
Andy Bavier1d631762022-01-10 15:47:51 -0800133 make 5g-test
Andy Bavierf73c3d22021-08-30 10:29:06 -0700134
Andy Bavier840e0b42021-09-16 17:10:23 -0700135By default the above commands install the 5G SD-CORE from the local charts in the Git repos cloned
136earlier. In order to install the SD-CORE using the latest published charts, add *CHARTS=latest*
137to the command, e.g.,::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700138
Andy Bavier1d631762022-01-10 15:47:51 -0800139 CHARTS=latest make 5g-test
Andy Bavierf73c3d22021-08-30 10:29:06 -0700140
Andy Bavier99892132022-03-11 14:49:10 -0700141To install the Aether 2.0 release, add *CHARTS=release-2.0*::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700142
Andy Bavier05eca6b2022-04-01 15:14:34 -0400143 CHARTS=release-2.0 make 5g-test
Andy Bavierf73c3d22021-08-30 10:29:06 -0700144
Andy Bavier1d631762022-01-10 15:47:51 -0800145To change the behavior of the test run by gNBSim, change the contents of *gnb.conf*
Andy Bavier4cda1402022-02-15 15:33:31 -0700146in *sd-core-5g-values.yaml*. Consult the
Andy Bavier1d631762022-01-10 15:47:51 -0800147`gNBSim documentation <https://docs.sd-core.opennetworking.org/master/developer/gnbsim.html>`_ for more information.
Andy Bavierf73c3d22021-08-30 10:29:06 -0700148
Andy Bavier4d27e4e2022-02-07 14:49:45 -0700149Exploring AiaB
150--------------
151
152The *kubectl* tool is the best way to get familiar with the pods and other Kubernetes objects installed by AiaB.
153The SD-CORE services, UPF, and simulated edge devices run in the *omec* namespace, while the ROC is running
154in the *aether-roc* namespace.
155
156The ROC GUI is available on port 31194 on the host running AiaB.
157
Andy Bavier840e0b42021-09-16 17:10:23 -0700158Cleanup
159-------
Andy Bavierf73c3d22021-08-30 10:29:06 -0700160
Andy Bavier840e0b42021-09-16 17:10:23 -0700161The first time you build AiaB, it takes a while because it sets up the Kubernetes cluster.
162Subsequent builds will be much faster if you follow these steps to clean up the Helm charts without
163destroying the Kubernetes cluster.
Andy Bavierf73c3d22021-08-30 10:29:06 -0700164
Andy Bavier840e0b42021-09-16 17:10:23 -0700165* Clean up the 4G SD-CORE: *make reset-test*
Andy Bavier72547672022-02-03 14:45:16 -0700166* Reset the 4G UE / eNB in order to re-run the 4G test: *make reset-ue*
Andy Bavier840e0b42021-09-16 17:10:23 -0700167* Clean up the 5G SD-CORE: *make reset-5g-test*
168* Clean up the ROC: *make roc-clean*
169
170It's normal for the above commands to take a minute or two to complete.
171
172As an example, suppose that you want to test the 4G SD-CORE with the ROC, and then the 5G SD-CORE
173with the ROC. You could run these commands::
174
175 CHARTS=latest make roc-4g-models # Install ROC with 4G configuration
176 CHARTS=latest make test # Install 4G SD-CORE and run ping test
177 make reset-test
178 make roc-clean
179 CHARTS=latest make roc-5g-models # Install ROC with 5G configuration
Andy Bavier1d631762022-01-10 15:47:51 -0800180 CHARTS=latest make 5g-test # Install 5G SD-CORE and run gNB Sim test
Andy Bavier840e0b42021-09-16 17:10:23 -0700181 make reset-5g-test
182 make roc-clean
183
Andy Bavier62678402022-04-18 12:06:52 -0700184To completely remove AiaB by tearing down the Kubernetes cluster, run *make clean*.
185
Andy Bavier840e0b42021-09-16 17:10:23 -0700186Developer Loop
187--------------
188
189Suppose you wish to test a new build of a 5G SD-CORE services. You can deploy custom images
Andy Bavier4cda1402022-02-15 15:33:31 -0700190by editing `~/aether-in-a-box/sd-core-5g-values.yaml`, for example::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700191
Andy Bavier4cda1402022-02-15 15:33:31 -0700192 omec-control-plane:
193 images:
194 tags:
195 webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f
196 pullPolicy: IfNotPresent
Andy Bavierf73c3d22021-08-30 10:29:06 -0700197
Andy Bavier840e0b42021-09-16 17:10:23 -0700198To upgrade a running 5G SD-CORE with the new image, or to deploy the 5G SD-CORE with the image::
Andy Bavierf73c3d22021-08-30 10:29:06 -0700199
Andy Bavier4cda1402022-02-15 15:33:31 -0700200 make reset-5g-test; make 5g-test
Andy Bavier840e0b42021-09-16 17:10:23 -0700201
202Troubleshooting / Known Issues
203------------------------------
204
205If you suspect a problem, first verify that all pods are in Running state::
206
207 kubectl -n omec get pods
208 kubectl -n aether-roc get pods
209
Andy Bavier840e0b42021-09-16 17:10:23 -07002104G Test Fails
211^^^^^^^^^^^^^
212Occasionally *make test* (for 4G) fails for unknown reasons; this is true regardless of which Helm charts are used.
Andy Bavier72547672022-02-03 14:45:16 -0700213If this happens, first try recreating the simulated UE / eNB and re-running the test as follows::
214
215 make reset-ue
216 make test
217
218If that does not work, try cleaning up AiaB as described above and re-building it.
219
220If *make test* fails consistently, check whether the configuration has been pushed to the SD-CORE::
Andy Bavier840e0b42021-09-16 17:10:23 -0700221
222 kubectl -n omec logs config4g-0 | grep "Successfully"
223
224You should see that a device group and slice has been pushed::
225
226 [INFO][WebUI][CONFIG] Successfully posted message for device group 4g-oaisim-user to main config thread
227 [INFO][WebUI][CONFIG] Successfully posted message for slice default to main config thread
228
229Then tail the *config4g-0* log and make sure that the configuration has been successfully pushed to all
230SD-CORE components.
Andy Bavier2e20f972022-04-21 11:09:02 -0700231
2325G Test Fails
233^^^^^^^^^^^^^
234
235If the 5G test fails (*make 5g-test*) then you will see output like this::
236
237 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Name: profile2 , Profile Type: pdusessest
238 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Ue's Passed: 2 , Ue's Failed: 3
239 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Profile Errors:
240 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007492, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
241 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007493, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
242 2022-04-21T17:59:12Z [ERRO][GNBSIM][Summary] imsi:imsi-208930100007494, procedure:REGISTRATION-PROCEDURE, error:triggering event:REGESTRATION-REQUEST-EVENT, expected event:AUTHENTICATION-REQUEST-EVENT, received event:REGESTRATION-REJECT-EVENT
243 2022-04-21T17:59:12Z [INFO][GNBSIM][Summary] Simulation Result: FAIL
244
245In this case check whether the *webui* pod has restarted... this can happen if it times out waiting
246for the database to come up::
247
248 $ kubectl -n omec get pod -l app=webui
249 NAME READY STATUS RESTARTS AGE
250 webui-6b9c957565-zjqls 1/1 Running 1 (6m55s ago) 7m56s
251
252If the output shows any restarts, then restart the *simapp* pod to cause it to re-push its subscriber state::
253
254 $ kubectl -n omec delete pod -l app=simapp
255 pod "simapp-6c49b87c96-hpf82" deleted
256
257Re-run the 5G test, it should now pass.