blob: 01711fc747e647be2552ba0780af141d0199bd3e [file] [log] [blame]
Andy Bavierf3e24b62022-03-30 19:53:01 -04001.. vim: syntax=rst
2
Scott Baker3a602662022-04-04 13:59:27 -07003Aether-in-a-Box on Hardware Radios
4==================================
Andy Bavierf3e24b62022-03-30 19:53:01 -04005
6This document describes how to set up an Aether-in-a-Box (AiaB) with
7a Sercomm eNodeB and connect real devices (e.g., 4G phones). This
8setup is suitable for laboratory experiments and proof-of-concept deployments.
9To create this setup you will need the following equipment:
10
11* Server for running AiaB (SD-CORE / UPF / ROC)
12
13 * Haswell CPU family or newer
14 * At least 4 CPUs and 12GB RAM
15 * Internet connection
16
Scott Baker3a602662022-04-04 13:59:27 -070017* 4G or 5G small cell eNodeB
Andy Bavierf3e24b62022-03-30 19:53:01 -040018
Scott Baker3a602662022-04-04 13:59:27 -070019 * Example: Sercomm CBRS LTE small cell eNodeB
20
21 * Firmware version 3918 or newer. For instructions on how to update the firmware see `here <https://docs.aetherproject.org/master/edge_deployment/enb_installation.html#upgrade-firmware>`_.
Andy Bavierf3e24b62022-03-30 19:53:01 -040022
23* SIM card writer and blank SIM cards
24
25We assume that the server and the eNodeB are connected to the same
26LAN, and the LAN also provides external Internet connectivity.
27
28Preparation
29-----------
30
31Create SIM cards by following the instructions for your SIM card writer.
32Of course you are free to use any values for IMSI, etc. that you choose,
33but these are the values that will work with the rest of the configuration
34in this document:
35
36* IMSI: each one is unique, matching pattern ``315010*********`` (15 digits)
37* OPc: ``69d5c2eb2e2e624750541d3bbc692ba5``
38* Transport Key: ``000102030405060708090a0b0c0d0e0f``
39
40If you choose different values for your SIM cards, you will need to
41modify subsequent configuration steps appropriately.
42
43Insert the SIM cards in devices that you wish to be able to connect to the Aether network.
44
45Server setup
46------------
47
48The server will run Aether-in-a-Box. The eNodeB will connect to the server over the local network.
49Perform these steps to prepare the server for the AiaB install:
50
51* Connect the server to the local network
52* Perform a clean install of Ubuntu 18.04 on the server
53* Set up password-less sudo for the user that will install Aether-in-a-Box
54
55After the steps above have been completed, install Aether-in-a-Box as follows::
56
57 sudo apt install git make
58 git clone "https://gerrit.opencord.org/aether-in-a-box"
59 cd aether-in-a-box
60
61Next, modify the file *sd-core-4g-values.yaml*. Under ``subscribers``,
62add an IMSI range for the SIM cards you created, with the Transport Key
63and OPc values you used earlier. For example, the following will add
64IMSIs between 315010999912301 and 315010999912303::
65
66 subscribers:
67 - ueId-start: 315010999912301
68 ueId-end: 315010999912303
69 plmnId: 315010
70 opc: 69d5c2eb2e2e624750541d3bbc692ba5
71 key: 000102030405060708090a0b0c0d0e0f
72 sequenceNumber: 135
73
74Determine which is the interface that has L3 connectivity to the
Andy Bavier5a32da02022-04-01 13:46:54 -040075eNodeB -- this will be ``DATA_IFACE`` in the configuration later. If
Andy Bavierf3e24b62022-03-30 19:53:01 -040076the eNodeB will also be connected to the local network, then this is just the
77servers primary interface. If the eNodeB will be connected via an
Andy Bavier5a32da02022-04-01 13:46:54 -040078isolated L2/L3 network segment, then ``DATA_IFACE`` refers to the server
Andy Bavierf3e24b62022-03-30 19:53:01 -040079interface on that network. Remember this interface for later.
80
81Option 1: Configure Aether with ROC
82^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
83
84The Aether ROC provides a GUI and API for dynamically configuring
85Aether. If you dont wish to use the ROC to configure AiaB, you
86can skip to the next section.
87
Andy Bavier5a32da02022-04-01 13:46:54 -040088Install AiaB as follows (specifying ``DATA_IFACE`` from above)::
Andy Bavierf3e24b62022-03-30 19:53:01 -040089
90 ENABLE_OAISIM=false DATA_IFACE=<iface> CHARTS=latest make roc-4g-models 4g-core
91
92Next, use the ROC to add information about your SIM cards.
93The ROC GUI is available at `http://<server-ip>:31194`.
94
Andy Bavier5a32da02022-04-01 13:46:54 -040095Choose ``Configuration > Site`` from the drop-down at top right and edit
96the ``AiaB site``. Change the following values and click ``Update``:
Andy Bavierf3e24b62022-03-30 19:53:01 -040097
98* MCC: 315
99* MNC: 010
100
Andy Bavier5a32da02022-04-01 13:46:54 -0400101Choose ``Sim Cards`` from the drop-down at top right. Edit the
Andy Bavierf3e24b62022-03-30 19:53:01 -0400102existing entries to reflect the SIM cards you are adding to devices
Andy Bavier5a32da02022-04-01 13:46:54 -0400103by replacing their IMSI values. Click ``Update`` after each edit.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400104If you want to connect more than two devices, consult the `ROC
105documentation <https://docs.aetherproject.org/master/operations/subscriber.html#configure-connectivity-service-for-a-new-device>`_.
106
Andy Bavier5a32da02022-04-01 13:46:54 -0400107Finally, click the Basket icon at top right and click the ``Commit`` button.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400108
109Now jump to the `Verifying the AiaB installation`_ section.
110
111Option 2: Configure Aether without ROC
112^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
113
114It is possible to configure Aether without the ROC,
115using static YAML files and the SimApp service. If you have already
116installed the ROC, you should skip this section.
117
118Edit *sd-core-4g-values.yaml*. Change ``mcc`` and ``mnc`` as follows::
119
120 plmn:
121 mcc: "315"
122 mnc: "010"
123
124Also add the IMSIs of your devices under ``imsis``, for example::
125
126 device-groups:
127 - name: "4g-oaisim-user"
128 imsis:
129 - "315010999912301"
130 - "315010999912302"
131 - "315010999912303"
132
Andy Bavier5a32da02022-04-01 13:46:54 -0400133Install AiaB as follows (specifying ``DATA_IFACE`` from above)::
Andy Bavierf3e24b62022-03-30 19:53:01 -0400134
135 ENABLE_OAISIM=false DATA_IFACE=<iface> CHARTS=latest make 4g-core
136
137Verifying the AiaB installation
138^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
139
140Installing AiaB will take about 20 minutes with a fast Internet
Andy Bavier5a32da02022-04-01 13:46:54 -0400141connection. If you see any errors / timeouts, try running the ``make``
Andy Bavierf3e24b62022-03-30 19:53:01 -0400142command again. The build will finish with a message:
143Your MME IP address is This is just the IP address assigned to
Andy Bavier5a32da02022-04-01 13:46:54 -0400144the ``DATA_IFACE``. Remember this for the eNodeB setup.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400145
146When the install is complete, check that the 4G SD-CORE is running
147as follows::
148
149 $ kubectl -n omec get pod
150 NAME READY STATUS RESTARTS AGE
151 cassandra-0 1/1 Running 0 7m27s
152 config4g-0 1/1 Running 0 7m27s
153 hss-0 1/1 Running 0 7m27s
154 mme-0 4/4 Running 0 7m27s
155 pcrf-0 1/1 Running 0 7m27s
156 simapp-65dc44b9d-stx6q 1/1 Running 0 7m27s
157 spgwc-0 2/2 Running 0 7m27s
158 upf-0 5/5 Running 0 7m27s
159
160You should see all pods in Running status.
161
162If you have installed the ROC, check that all its pods are running
163as follows::
164
165 $ kubectl -n aether-roc get pod
166 NAME READY STATUS RESTARTS AGE
167 aether-roc-api-78cc548bb9-7vjs2 1/1 Running 0 4m16s
168 aether-roc-gui-v2-6d674fd446-tttb5 1/1 Running 0 4m16s
169 aether-roc-umbrella-grafana-74f8489c8f-s9p45 2/2 Running 0 4m16s
170 aether-roc-websocket-855d64549b-44fnc 1/1 Running 0 4m16s
171 onos-cli-5d448ff6c4-stq5t 1/1 Running 0 4m16s
172 onos-config-7f4df96b88-vtp5s 6/6 Running 0 4m16s
173 onos-consensus-store-0 1/1 Running 0 4m15s
174 onos-topo-585c7c8976-6jq7b 3/3 Running 0 4m16s
175 sdcore-adapter-v2-5646d455b9-2d6zl 1/1 Running 0 4m15s
176
177You should see all pods in Running status.
178
Andy Bavier62678402022-04-18 12:06:52 -0700179The UPF pod connects to the ``DATA_IFACE`` specified above using macvlan networks called
180``core`` and ``access``. Next, check that these have been successfully created, e.g. using
181``ifconfig``::
182
183 $ ifconfig core
184 core: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
185 inet 192.168.250.1 netmask 255.255.255.0 broadcast 192.168.250.255
186 ether 16:9d:c1:0f:19:3a txqueuelen 1000 (Ethernet)
187 RX packets 513797 bytes 48400525 (48.4 MB)
188 RX errors 0 dropped 0 overruns 0 frame 0
189 TX packets 102996 bytes 26530538 (26.5 MB)
190 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
191
192 $ ifconfig access
193 access: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
194 inet 192.168.252.1 netmask 255.255.255.0 broadcast 192.168.252.255
195 ether 7a:9f:38:c0:18:15 txqueuelen 1000 (Ethernet)
196 RX packets 558162 bytes 64064410 (64.0 MB)
197 RX errors 0 dropped 0 overruns 0 frame 0
198 TX packets 99553 bytes 16646682 (16.6 MB)
199 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
200
Andy Bavierf3e24b62022-03-30 19:53:01 -0400201Sercomm eNodeB setup
202--------------------
203
204The instructions in this section describe a basic configuration of
205the eNodeB. For a more comprehensive guide to
206eNodeB configuration see `eNB Installation <https://docs.aetherproject.org/master/edge_deployment/enb_installation.html>`_.
207
208The Sercomm eNodeB has two Ethernet ports: WAN and LAN. We will use
209the LAN port for configuration of the eNodeB and the WAN port for
210normal operation. Connect the eNodeB WAN port to the local network.
211
212Connect the eNodeB LAN port to a free Ethernet port on a Linux machine
213(say, a laptop) that will be used for the initial configuration of
Andy Bavier5a32da02022-04-01 13:46:54 -0400214the eNodeB. On that machine run ``dhclient`` on the interface corresponding
Andy Bavierf3e24b62022-03-30 19:53:01 -0400215to the Ethernet port, for example::
216
217 sudo dhclient eth1
218
219The interface should receive an IP address from the Sercomm eNodeB on
Andy Bavier5a32da02022-04-01 13:46:54 -0400220the 11.11.11.0/24 subnet. Check this using ``ifconfig``::
Andy Bavierf3e24b62022-03-30 19:53:01 -0400221
222 $ ifconfig eth1
223 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
224 inet 11.11.11.100 netmask 255.255.255.0 broadcast 11.11.11.255
225 inet6 fe80::2e0:4cff:fe68:2f76 prefixlen 64 scopeid 0x20<link>
226 ether 00:e0:4c:68:2f:76 txqueuelen 1000 (Ethernet)
227 RX packets 264652 bytes 216094312 (216.0 MB)
228 RX errors 0 dropped 0 overruns 0 frame 0
229 TX packets 183978 bytes 36528580 (36.5 MB)
230 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
231
232The eNodeB IP address should be 11.11.11.188 on that link. Verify
233this using ping::
234
235 $ ping 11.11.11.188
236
237To access the Sercomm eNodeB admin page, you can run a Web browser on
238the laptop and direct it to `https://11.11.11.188`. Login to the admin
Andy Bavier5a32da02022-04-01 13:46:54 -0400239interface as user: ``sc_femto`` and password: ``scHt3pp``.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400240
241Click the ``NetWork set`` menu at the top. In the first section
Andy Bavier5a32da02022-04-01 13:46:54 -0400242``IP Address``, set *Connect type: DHCP* (assuming this is supported on
Andy Bavierf3e24b62022-03-30 19:53:01 -0400243your local network, otherwise set up a static IP address).
244Click the blue ``Save`` button at the bottom.
245
246Next, click the ``Manage`` menu at the top and then click the
247``LTE Basic Setting`` tab. Change these parameters and click ``Save``:
248
249* Carrier Number: 2
250* Carrier Aggregation: Unset
251* BandWidth: 20
252* FrequencyBand: 48,48
253* EARFCN: 55440,55640
254* CellIdentity: 2,1
255* PCI: 100,101
256* TxPower: 20
257* Tunnel Type: IPv4
258* MME IP Address: <MME IP address from AiaB installation>
259* PLMNID: 315010
260* TAC: 1
261* Sync Source: FREE_RUNNING
262* Enable CWMP: Unset
263
264Click the ``SAS Configuration`` tab. In the ``Location Configuration``
265section, enter these values and click Save”:
266
267* Location: Indoor
268* Location Source: Manual
269* Latitude: 0
270* Longitude: 0
271* Elevation: -18000
272
273Next we need to add a static route to the UPF address, 192.168.252.3,
274on the eNodeB. Click on ``TR098`` menu and then click on ``IP`` tab.
275Scroll down to ``Device.Routing.Router.`` section. Click ``View List``.
276Add this info on the blank line:
277
278* Enable: Set
279* StaticRoute: Set
280* DestIPAddress: 192.168.252.0
281* DestSubnetMask: 255.255.255.0
282* GatewayIPAddress: <Use MME IP address from AiaB installation>
283* Interface: Device.IP.Interface.1.
284
Andy Bavier5a32da02022-04-01 13:46:54 -0400285Then click the ``Add`` button at the far right.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400286
287Finally click the ``FAPService`` menu and then go to the ``FAPControl``
288tab. Check the box next to ``AdminState`` in the first section and
289click ``Save``.
290
291After these changes are made, reboot the eNodeB by clicking the red
292``power button`` square at top right and selecting ``Reboot``. When the
293eNodeB comes back up, it should have an IP address on the network
294(via the WAN port), and the admin page should now be available on
295`https://<endoeb-ip>`.
296
297Test connectivity from the eNodeB to the MME and the UPF running on
298the server as follows. Login to the eNodeB admin interface, click
299the Manage menu at the top, and click the ``IP Diagnose`` tab. Under
300``Ping and Traceroute``, select ``ping``, and then type the following IP
301addresses into the box to the right and click ``Run``:
302
303* <MME IP address from AiaB installation>
304* 192.168.251.1
305* 192.168.252.3
306
307If all of these are working, then you are ready to try to connect
308devices to the network.
309
310Connecting Devices
311------------------
312
Andy Bavier6f431e42022-04-01 13:52:22 -0400313Documenting how to configure different types of devices to work
314with Aether is work-in-progress.
Andy Bavierf3e24b62022-03-30 19:53:01 -0400315
Andy Bavierc757e612022-05-24 16:13:51 -0700316Enable Monitoring
317-----------------
318
319To easily monitor your AiaB deployment, you can run the following command to
320deploy the Prometheus/Grafana monitoring stack on your AiaB server::
321
322 make monitoring-4g
323
324After this step, Grafana is available at `http://<server-ip>:30950`. You will see a number of system dashboards
325for monitoring Kubernetes, as well as a simple AiaB dashboard that enables inspection of the local Aether state.
326
327.. figure:: images/4g-aiab-grafana-dashboard.png
328 :align: center
329 :width: 100 %
330
331 *4G Grafana AiaB Dashboard*
332
333The dashboard shows whether the eNodeB is connected to the core, how many active UEs there are, and the
334uplink (Tx Bitrate) and downlink (Rx Bitrate) throughput at the UPF.
335
Andy Bavier62678402022-04-18 12:06:52 -0700336Troubleshooting
337---------------
Andy Bavierf3e24b62022-03-30 19:53:01 -0400338
Andy Bavier62678402022-04-18 12:06:52 -0700339AiaB connects macvlan networks to ``DATA_IFACE`` so that the UPF can communicate on the network.
340To do this it assumes that the *systemd-networkd* service is installed and running, ``DATA_IFACE``
341is under its control, and the systemd-networkd configuration file for ``DATA_IFACE`` ends with
342``<DATA_IFACE>.network``, where ``<DATA_IFACE>`` stands for the actual interface name. It
343tries to find this configuration file by looking in the standard paths. If it fails you'll see
344a message like::
345
346 FATAL: Could not find systemd-networkd config for interface foobar, exiting now!
347 make: *** [Makefile:112: /users/acb/aether-in-a-box//build/milestones/interface-check] Error 1
348
349In this case, you can specify a ``DATA_IFACE_PATH=<path to the config file>`` argument to ``make``
350so that AiaB can find the systemd-networkd configuration file for ``DATA_IFACE``.
351
352Restarting the AiaB Server
353--------------------------
354
355AiaB should come up in a mostly working state if the AiaB server is rebooted. If any pods are
356stuck in an Error or CrashLoopBackoff state they can be restarted using ``kubectl delete pod``.
357It might also be necessary to power cycle the Sercomm eNodeB in order to get it to reconnect to
358the SD-CORE.