Larry Peterson | def1b67 | 2023-08-07 14:06:24 -0700 | [diff] [blame] | 1 | Physical RAN |
| 2 | --------------- |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 3 | |
| 4 | We are now ready to replace the emulated RAN with physical gNBs and |
| 5 | real UEs. You will need to edit ``hosts.ini`` to reflect the Aether |
| 6 | cluster you want to support, where just a single server is sufficient |
| 7 | and there is no reason to include nodes in the ``[gnbsim_nodes]`` set. |
| 8 | We also assume you start with a variant of ``vars/main.yml`` |
| 9 | customized for running physical 5G radios, which is easy to do: |
| 10 | |
| 11 | .. code-block:: |
| 12 | |
| 13 | $ cd vars |
| 14 | $ cp main-gNB.yml main.yml |
| 15 | |
| 16 | The following focuses on a single gNB, which we assume is connected to |
| 17 | the same L2 network as the Aether cluster. In our running example, |
| 18 | this implies both are on subnet ``10.76.28.0/24``. |
| 19 | |
Larry Peterson | 8b1ed83 | 2023-08-29 12:06:06 -0700 | [diff] [blame] | 20 | .. admonition:: Troubleshooting Hint |
| 21 | |
| 22 | Physical gNBs connect to SD-Core (both the AMF in the control plane |
| 23 | and the UPF in the user plane) in exactly the same way as external |
| 24 | instances of gNBsim. Going through the process of bringing up gNBsim |
| 25 | in a second server, as described in the previous section, is a good |
| 26 | way to validate that your Core is "gNB-ready". |
| 27 | |
| 28 | |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 29 | Modify Configuration |
| 30 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 31 | |
| 32 | Modify the ``core`` section of ``vars/main.yml`` to match the |
| 33 | following, substituting your local details for ``ens18`` and |
| 34 | ``10.76.28.113``. Of particular note, setting ``ran_subnet`` to the |
| 35 | empty string indicates that the gNB is connected to the same physical |
| 36 | L2 network as your Aether cluster, and the new ``values_file`` is |
| 37 | tailored for a physical RAN rather than the emulated RAN we've been |
| 38 | using. |
| 39 | |
| 40 | .. code-block:: |
| 41 | |
| 42 | core: |
| 43 | standalone: "true" |
| 44 | data_iface: ens18 |
| 45 | values_file: "deps/5gc/roles/core/templates/radio-5g-values.yaml" |
| 46 | ran_subnet: "" |
| 47 | helm: |
| 48 | chart_ref: aether/sd-core |
| 49 | chart_version: 0.12.6 |
| 50 | upf: |
| 51 | ip_prefix: "192.168.252.0/24" |
| 52 | amf: |
| 53 | ip: "10.76.28.113" |
| 54 | |
| 55 | |
| 56 | Prepare UEs |
| 57 | ~~~~~~~~~~~~ |
| 58 | |
| 59 | 5G-connected devices must have a SIM card, which you are responsible |
| 60 | for creating and inserting. You will need a SIM card writer (these |
| 61 | are readily available for purchase on Amazon) and a PLMN identifier |
| 62 | constructed from a valid MCC/MNC pair. For our purposes, we use two |
| 63 | different PLMN ids: ``315010`` constructed from MCC=315 (US) and |
| 64 | MNC=010 (CBRS), and ``00101`` constructed from MCC=001 (TEST) and |
| 65 | MNC=01 (TEST). You should use whatever values are appropriate for your |
| 66 | local environment. You then assign an IMSI and two secret keys to |
| 67 | each SIM card. Throughout this section, we use the following values as |
| 68 | exemplars: |
| 69 | |
| 70 | * IMSI: each one is unique, matching pattern ``315010*********`` (up to 15 digits) |
| 71 | * OPc: ``69d5c2eb2e2e624750541d3bbc692ba5`` |
| 72 | * Key: ``000102030405060708090a0b0c0d0e0f`` |
| 73 | |
Larry Peterson | 505d4b3 | 2023-08-21 14:39:45 -0700 | [diff] [blame] | 74 | Note that the actual config files distributed with OnRamp have IMSIs |
| 75 | constructed using PLMN id ``00101``. Both sets of examples are taken |
Larry Peterson | 5d6b3b3 | 2023-09-05 15:11:55 -0700 | [diff] [blame^] | 76 | from working deployments (``315010`` for a 4G/eNB and ``00101`` for a |
| 77 | 5G/gNB), so in principle either should work as a model you can emulate |
| 78 | in your deployment. As a practical matter, however, it is certainly |
| 79 | easiest (and safest) to start with the existing code. |
Larry Peterson | 505d4b3 | 2023-08-21 14:39:45 -0700 | [diff] [blame] | 80 | |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 81 | Insert the SIM cards into whatever devices you plan to connect to |
| 82 | Aether. Be aware that not all phones support the CBRS frequency bands |
Larry Peterson | 5d6b3b3 | 2023-09-05 15:11:55 -0700 | [diff] [blame^] | 83 | that Aether uses, specifically n48 and n78. Aether is known to work |
| 84 | with recent iPhones (11 and greater), Google Pixel phones (4 and |
| 85 | greater), OnePlus phones, and Moto G 5G phones for band n78. Aether |
| 86 | is known to work with Moto G 5G and OnePlus phones for band n48; Pixel |
| 87 | 7 phones are purported to work as well. Note that on each phone you |
| 88 | will need to configure ``internet`` as the *Access Point Name (APN)*. |
| 89 | Another option is to use a 5G dongle connected to a Raspberry Pi as a |
| 90 | demonstration UE. This makes it easier to run diagnostic tests from |
| 91 | the UE. For example, we have used `APAL's 5G dongle |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 92 | <https://www.apaltec.com/dongle/>`__ with Aether. |
| 93 | |
| 94 | Finally, modify the ``subscribers`` block of the |
| 95 | ``omec-sub-provision`` section in file |
| 96 | ``deps/5gc/roles/core/templates/radio-5g-values.yaml`` to record the IMSI, |
| 97 | OPc, and Key values configured onto your SIM cards. The block also |
| 98 | defines a sequence number that is intended to thwart replay |
| 99 | attacks. For example, the following code block adds IMSIs between |
| 100 | ``315010999912301`` and ``315010999912310``: |
| 101 | |
| 102 | .. code-block:: |
| 103 | |
| 104 | subscribers: |
| 105 | - ueId-start: "315010999912301" |
| 106 | ueId-end: "315010999912310" |
| 107 | plmnId: "315010" |
| 108 | opc: "69d5c2eb2e2e624750541d3bbc692ba5" |
| 109 | key: "000102030405060708090a0b0c0d0e0f" |
| 110 | sequenceNumber: 135 |
| 111 | |
| 112 | Further down in the same ``omec-sub-provision`` section you will find |
| 113 | two other blocks that also need to be edited. The first, |
| 114 | ``device-groups``, assigns IMSIs to *Device Groups*. You will need to |
| 115 | reenter the individual IMSIs from the ``subscribers`` block that will |
| 116 | be part of the device-group: |
| 117 | |
| 118 | .. code-block:: |
| 119 | |
| 120 | device-groups: |
| 121 | - name: "5g-user-group1" |
| 122 | imsis: |
| 123 | - "315010999912301" |
| 124 | - "315010999912302" |
| 125 | - "315010999912303" |
| 126 | |
| 127 | The second block, ``network-slices``, sets various parameters |
| 128 | associated with the *Slices* that connect device groups to |
| 129 | applications. Here, you will need to reenter the PLMN information, |
| 130 | with the other slice parameters remaining unchanged (for now): |
| 131 | |
| 132 | .. code-block:: |
| 133 | |
| 134 | plmn: |
| 135 | mcc: "315" |
| 136 | mnc: "010" |
| 137 | |
| 138 | Aether supports multiple *Device Groups* and *Slices*, but the data |
| 139 | entered here is purposely minimal; it's just enough to bring up and |
| 140 | debug the installation. Over the lifetime of a running system, |
| 141 | information about *Device Groups* and *Slices* (and the other |
| 142 | abstractions they build upon) should be entered via the ROC, as |
| 143 | described the section on Runtime Control. When you get to that point, |
| 144 | Ansible variable ``standalone`` in ``vars/main.yml`` (which |
| 145 | corresponds to the override value assigned to |
| 146 | ``provision-network-slice`` in ``radio-5g-values.yaml``) should be set |
| 147 | to ``false``. Doing so causes the ``device-groups`` and |
| 148 | ``network-slices`` blocks of ``radio-5g-values.yaml`` to be |
| 149 | ignored. The ``subscribers`` block is always required to configure |
| 150 | SD-Core. |
| 151 | |
| 152 | |
| 153 | Bring Up Aether |
| 154 | ~~~~~~~~~~~~~~~~~~~~~ |
| 155 | |
| 156 | You are now ready to bring Aether on-line. We assume a fresh install |
Larry Peterson | def1b67 | 2023-08-07 14:06:24 -0700 | [diff] [blame] | 157 | by typing the following: |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 158 | |
| 159 | .. code-block:: |
| 160 | |
| 161 | $ make aether-k8s-install |
| 162 | $ make aether-5gc-install |
| 163 | |
| 164 | You can verify the installation by running ``kubectl`` just as you did |
| 165 | in earlier stages. Note that we postpone bringing up the AMP until |
| 166 | later so as to have fewer moving parts to debug. |
| 167 | |
| 168 | |
| 169 | gNodeB Setup |
| 170 | ~~~~~~~~~~~~~~~~~~~~ |
| 171 | |
| 172 | Once the SD-Core is up and running, we are ready to bring up the |
| 173 | physical gNB. The details of how to do this depend on the specific |
| 174 | device you are using, but we identify the main issues you need to |
Larry Peterson | 5d6b3b3 | 2023-09-05 15:11:55 -0700 | [diff] [blame^] | 175 | address using SERCOMM's 5G femto cell (as distributed by MosoLabs) as |
| 176 | an example. That particular device uses either the n48 or n78 band and |
| 177 | is on the ONF MarketPlace, where you can also find a User's Guide. |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 178 | |
| 179 | .. _reading_sercomm: |
| 180 | .. admonition:: Further Reading |
| 181 | |
Larry Peterson | 5d6b3b3 | 2023-09-05 15:11:55 -0700 | [diff] [blame^] | 182 | `MOSO CANOPY 5G INDOOR SMALL CELL |
| 183 | <https://opennetworking.org/products/moso-canopy-5g-indoor-small-cell/>`__. |
| 184 | |
| 185 | .. admonition:: Troubleshooting Hint |
| 186 | |
| 187 | The product data sheet shows support for frequency bands |
| 188 | n78/n48/n77, but individual devices do not necessarily support all |
| 189 | three. For example, we have experience with an n78 device and an n48 |
| 190 | device, with the latter (n48) now becoming the default. For that |
| 191 | band, PLMN id ``00101`` is currently recommended. |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 192 | |
| 193 | For the purposes of the following description, we assume the gNB is |
| 194 | assigned IP address ``10.76.28.187``, which per our running example, |
| 195 | is on the same L2 network as our Aether server (``10.76.28.113``). |
Larry Peterson | def1b67 | 2023-08-07 14:06:24 -0700 | [diff] [blame] | 196 | :numref:`Figure %s <fig-sercomm>` shows a screenshot of the SERCOMM |
llp | b353464 | 2023-08-02 09:23:52 -0700 | [diff] [blame] | 197 | gNB management dashboard, which we reference in the instructions that |
| 198 | follow: |
| 199 | |
| 200 | .. _fig-sercomm: |
| 201 | .. figure:: figures/Sercomm.png |
| 202 | :width: 500px |
| 203 | :align: center |
| 204 | |
| 205 | Management dashboard on the Sercomm gNB, showing the dropdown |
| 206 | ``Settings`` menu overlayed on the ``NR Cell Configuration`` page |
| 207 | (which shows default radio settings). |
| 208 | |
| 209 | |
| 210 | 1. **Connect to Management Interface.** Start by connecting a laptop |
| 211 | directly to the LAN port on the small cell, pointing your laptop's |
| 212 | web browser at the device's management page |
| 213 | (``https://10.10.10.189``). You will need to assign your laptop an |
| 214 | IP address on the same subnet (e.g., ``10.10.10.100``). Once |
| 215 | connected, log in with the credentials provided by the vendor. |
| 216 | |
| 217 | 2. **Configure WAN.** Visit the ``Settings > WAN`` page to configure |
| 218 | how the small cell connects to the Internet via its WAN port, |
| 219 | either dynamically using DHCP or statically by setting the device's |
| 220 | IP address (``10.76.28.187``) and default gateway (``10.76.28.1``). |
| 221 | |
| 222 | 3. **Access Remote Management.** Once on the Internet, it should be |
| 223 | possible to reach the management dashboard without being directly |
| 224 | connected to the LAN port (``https://10.76.28.187``). |
| 225 | |
| 226 | 4. **Connect GPS.** Connect the small cell's GPS antenna to the GPS |
| 227 | port, and place the antenna so it has line-of-site to the sky |
| 228 | (i.e., place it in a window). The ``Status`` page of the management |
| 229 | dashboard should report its latitude, longitude, and fix time. |
| 230 | |
| 231 | 5. **Spectrum Access System.** One reason the radio needs GPS is so it |
| 232 | can report its location to a Spectrum Access System (SAS), a |
| 233 | requirement in the US to coordinate access to the CBRS Spectrum in |
| 234 | the 3.5 GHz band. For example, the production deployment of Aether |
| 235 | uses the `Google SAS portal |
| 236 | <https://cloud.google.com/spectrum-access-system/docs/overview>`__, |
| 237 | which the small cell can be configured to query periodically. To do |
| 238 | so, visit the ``Settings > SAS`` page. Acquiring the credentials |
| 239 | needed to access the SAS requires you go through a certification |
| 240 | process, but as a practical matter, it may be possible to test an |
| 241 | isolated/low-power femto cell indoors before completing that |
| 242 | process. Consult with your local network administrator. |
| 243 | |
| 244 | 6. **Configure Radio Parameters.** Visit the ``Settings > NR Cell |
| 245 | Configuration`` page (shown in the figure) to set parameters that |
| 246 | control the radio. It should be sufficient to use the default |
| 247 | settings when getting started. |
| 248 | |
| 249 | 7. **Configure the PLMN.** Visit the ``Settings > 5GC`` page to set |
| 250 | the PLMN identifier on the small cell (``00101``) to match the |
| 251 | MCC/MNC values (``001`` / ``01`` ) specified in the Core. |
| 252 | |
| 253 | 8. **Connect to Aether Control Plane.** Also on the ``Settings > 5GC`` |
| 254 | page, define the AMF Address to be the IP address of your Aether |
| 255 | server (e.g., ``10.76.28.113``). Aether's SD-Core is configured to |
| 256 | expose the corresponding AMF via a well-known port, so the server's |
| 257 | IP address is sufficient to establish connectivity. The ``Status`` |
| 258 | page of the management dashboard should confirm that control |
| 259 | interface is established. |
| 260 | |
| 261 | 9. **Connect to Aether User Plane.** As described in an earlier |
| 262 | section, the Aether User Plane (UPF) is running at IP address |
| 263 | ``192.168.252.3``. Connecting to that address requires installing a |
| 264 | route to subnet ``192.168.252.0/24``. How you install this route is |
| 265 | device and site-dependent. If the small cell provides a means to |
| 266 | install static routes, then a route to destination |
| 267 | ``192.168.252.0/24`` via gateway ``10.76.28.113`` (the server |
| 268 | hosting Aether) will work. If the small cell does not allow static |
| 269 | routes (as is the case for the SERCOMM gNB), then ``10.76.28.113`` |
| 270 | can be installed as the default gateway, but doing so requires that |
| 271 | your server also be configured to forward IP packets on to the |
| 272 | Internet. |
| 273 | |
| 274 | Run Diagnostics |
| 275 | ~~~~~~~~~~~~~~~~~ |
| 276 | |
| 277 | Successfully connecting a UE to the Internet is not a straightforward |
| 278 | exercise. It involves configuring the UE, gNB, and SD-Core software in |
| 279 | a consistent way; establishing SCTP-based control plane (N2) and |
| 280 | GTP-based user plane (N3) connections between the base station and |
| 281 | Mobile Core; and traversing multiple IP subnets along the end-to-end |
| 282 | path. |
| 283 | |
| 284 | The UE and gNB provide limited diagnostic tools. For example, it's |
| 285 | possible to run ``ping`` and ``traceroute`` from both. You can also |
| 286 | run the ``ksniff`` tool described in the Networking section, but the |
| 287 | most helpful packet traces you can capture are shown in the following |
| 288 | commands. You can run these on the Aether server, where we use our |
| 289 | example ``ens18`` interface for illustrative purposes: |
| 290 | |
| 291 | .. code-block:: |
| 292 | |
| 293 | $ sudo tcpdump -i any sctp -w sctp-test.pcap |
| 294 | $ sudo tcpdump -i ens18 port 2152 -w gtp-outside.pcap |
| 295 | $ sudo tcpdump -i access port 2152 -w gtp-inside.pcap |
| 296 | $ sudo tcpdump -i core net 172.250.0.0/16 -w n6-inside.pcap |
| 297 | $ sudo tcpdump -i ens18 net 172.250.0.0/16 -w n6-outside.pcap |
| 298 | |
| 299 | The first trace, saved in file ``sctp.pcap``, captures SCTP packets |
| 300 | sent to establish the control path between the base station and the |
| 301 | Mobile Core (i.e., N2 messages). Toggling "Mobile Data" on the UE, |
| 302 | for example by turning Airplane Mode off and on, will generate the |
| 303 | relevant control plane traffic. |
| 304 | |
| 305 | The second and third traces, saved in files ``gtp-outside.pcap`` and |
| 306 | ``gtp-inside.pcap``, respectively, capture GTP packets (tunneled |
| 307 | through port ``2152`` ) on the RAN side of the UPF. Setting the |
| 308 | interface to ``ens18`` corresponds to "outside" the UPF and setting |
| 309 | the interface to ``access`` corresponds to "inside" the UPF. Running |
| 310 | ``ping`` from the UE will generate the relevant user plane (N3) traffic. |
| 311 | |
| 312 | Similarly, the fourth and fifth traces, saved in files |
| 313 | ``n6-inside.pcap`` and ``n6-outside.pcap``, respectively, capture IP |
| 314 | packets on the Internet side of the UPF (which is known as the **N6** |
| 315 | interface in 3GPP). In these two tests, ``net 172.250.0.0/16`` |
| 316 | corresponds to the IP addresses assigned to UEs by the SMF. Running |
| 317 | ``ping`` from the UE will generate the relevant user plane traffic. |
| 318 | |
| 319 | If the ``gtp-outside.pcap`` has packets and the ``gtp-inside.pcap`` |
| 320 | is empty (no packets captured), you may run the following commands |
| 321 | to make sure packets are forwarded from the ``ens18`` interface |
| 322 | to the ``access`` interface and vice versa: |
| 323 | |
| 324 | .. code-block:: |
| 325 | |
| 326 | $ sudo iptables -A FORWARD -i ens18 -o access -j ACCEPT |
| 327 | $ sudo iptables -A FORWARD -i access -o ens18 -j ACCEPT |
Larry Peterson | def1b67 | 2023-08-07 14:06:24 -0700 | [diff] [blame] | 328 | |
| 329 | Support for eNBs |
| 330 | ~~~~~~~~~~~~~~~~~~ |
| 331 | |
| 332 | Aether OnRamp is geared towards 5G, but it does support physical eNBs, |
| 333 | including 4G-based versions of both SD-Core and AMP. It does not |
| 334 | support an emulated 4G RAN. The 4G scenario uses all the same Ansible |
| 335 | machinery outlined in earlier sections, but uses a variant of |
| 336 | ``vars/main.yml`` customized for running physical 4G radios: |
| 337 | |
| 338 | .. code-block:: |
| 339 | |
| 340 | $ cd vars |
| 341 | $ cp main-eNB.yml main.yml |
| 342 | |
| 343 | Assuming that starting point, the following outlines the key |
| 344 | differences from the 5G case: |
| 345 | |
| 346 | 1. There is a 4G-specific repo, which you can find in ``deps/4gc``. |
| 347 | |
| 348 | 2. The ``core`` section of ``vars/main.yml`` specifies a 4G-specific values file: |
| 349 | |
| 350 | ``values_file: "deps/4gc/roles/core/templates/radio-4g-values.yaml"`` |
| 351 | |
| 352 | 3. The ``amp`` section of ``vars/main.yml`` specifies that 4G-specific |
| 353 | models and dashboards get loaded into the ROC and Monitoring |
| 354 | services, respectively: |
| 355 | |
| 356 | ``roc_models: "deps/amp/roles/roc-load/templates/roc-4g-models.json"`` |
| 357 | |
| 358 | ``monitor_dashboard: "deps/amp/roles/monitor-load/templates/4g-monitor"`` |
| 359 | |
| 360 | 4. You need to edit two files with details for the 4G SIM cards you |
| 361 | use. One is the 4G-specific values file used to configure SD-Core: |
| 362 | |
| 363 | ``deps/4gc/roles/core/templates/radio-4g-values.yaml`` |
| 364 | |
| 365 | The other is the 4G-specific Models file used to bootstrap ROC: |
| 366 | |
| 367 | ``deps/amp/roles/roc-load/templates/radio-4g-models.json`` |
| 368 | |
| 369 | 5. There are 4G-specific Make targets for SD-Core (e.g., ``make |
| 370 | aether-4gc-install`` and ``make aether-4gc-uninstall``), but the |
| 371 | Make targets for AMP (e.g., ``make aether-amp-install`` and ``make |
| 372 | aether-amp-uninstall``) work unchanged in both 4G and 5G. |
| 373 | |
| 374 | The Quick Start and Emulated RAN (gNBsim) deployments are for 5G only, |
| 375 | but revisiting the other sections—substituting the above for their 5G |
| 376 | counterparts—serves as a guide for deploying a 4G version of Aether. |
| 377 | Note that the network is configured in exactly the same way for both |
| 378 | 4G and 5G. This is because SD-Core's implementation of the UPF is used |
| 379 | in both cases. |