developer/aiabhw.rst

.. vim: syntax=rst

Aether-in-a-Box on Hardware Radios
==================================

This document describes how to set up an Aether-in-a-Box (AiaB) with
a Sercomm eNodeB and connect real devices (e.g., 4G phones).  This
setup is suitable for laboratory experiments and proof-of-concept deployments.
To create this setup you will need the following equipment:

* Server for running AiaB (SD-CORE / UPF / ROC)

  * Haswell CPU family or newer
  * At least 4 CPUs and 12GB RAM
  * Internet connection

* 4G or 5G small cell eNodeB

  * Example: Sercomm CBRS LTE small cell eNodeB

    * Firmware version 3918 or newer.  For instructions on how to update the firmware see `here <https://docs.aetherproject.org/master/edge_deployment/enb_installation.html#upgrade-firmware>`_.

* SIM card writer and blank SIM cards

We assume that the server and the eNodeB are connected to the same
LAN, and the LAN also provides external Internet connectivity.

Preparation
-----------

Create SIM cards by following the instructions for your SIM card writer.
Of course you are free to use any values for IMSI, etc. that you choose,
but these are the values that will work with the rest of the configuration
in this document:

* IMSI: each one is unique, matching pattern ``315010*********`` (15 digits)
* OPc: ``69d5c2eb2e2e624750541d3bbc692ba5``
* Transport Key: ``000102030405060708090a0b0c0d0e0f``

If you choose different values for your SIM cards, you will need to
modify subsequent configuration steps appropriately.

Insert the SIM cards in devices that you wish to be able to connect to the Aether network.

Server setup
------------

The server will run Aether-in-a-Box.  The eNodeB will connect to the server over the local network.
Perform these steps to prepare the server for the AiaB install:

* Connect the server to the local network
* Perform a clean install of Ubuntu 18.04 or Ubuntu 20.04 on the server
* Set up password-less sudo for the user that will install Aether-in-a-Box

After the steps above have been completed, install Aether-in-a-Box as follows::

    sudo apt install git make
    git clone "https://gerrit.opencord.org/aether-in-a-box"
    cd aether-in-a-box

Next, modify the file *sd-core-4g-values.yaml*.  Under ``subscribers``,
add an IMSI range for the SIM cards you created, with the Transport Key
and OPc values you used earlier.  For example, the following will add
IMSIs between 315010999912301 and 315010999912303::

    subscribers:
    - ueId-start: 315010999912301
      ueId-end: 315010999912303
      plmnId: 315010
      opc: 69d5c2eb2e2e624750541d3bbc692ba5
      key: 000102030405060708090a0b0c0d0e0f
      sequenceNumber: 135

Determine which is the interface that has L3 connectivity to the
eNodeB -- this will be ``DATA_IFACE`` in the configuration later.  If
the eNodeB will also be connected to the local network, then this is just the
server’s primary interface.  If the eNodeB will be connected via an
isolated L2/L3 network segment, then ``DATA_IFACE`` refers to the server
interface on that network.   Remember this interface for later.

Option 1: Configure Aether with ROC
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Aether ROC provides a GUI and API for dynamically configuring
Aether.  If you don’t wish to use the ROC to configure AiaB, you
can skip to the next section.

Install AiaB as follows (specifying ``DATA_IFACE`` from above)::

    ENABLE_OAISIM=false DATA_IFACE=<iface> CHARTS=latest make roc-4g-models 4g-core

Next, use the ROC to add information about your SIM cards.
The ROC GUI  is available at `http://<server-ip>:31194`.

Choose ``Configuration > Site`` from the drop-down at top right and edit
the ``AiaB site``.  Change the following values and click ``Update``:

* MCC: 315
* MNC: 010

Choose ``Sim Cards`` from the drop-down at top right.  Edit the
existing entries to reflect the SIM cards you are adding to devices
by replacing their IMSI values.  Click ``Update`` after each edit.
If you want to connect more than two devices, consult the `ROC
documentation <https://docs.aetherproject.org/master/operations/subscriber.html#configure-connectivity-service-for-a-new-device>`_.

Finally, click the Basket icon at top right and click the ``Commit`` button.

Now jump to the `Verifying the AiaB installation`_ section.

Option 2: Configure Aether without ROC
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is possible to configure Aether without the ROC,
using static YAML files and the SimApp service.  If you have already
installed the ROC, you should skip this section.

Edit *sd-core-4g-values.yaml*.  Change ``mcc`` and ``mnc`` as follows::

    plmn:
      mcc: "315"
      mnc: "010"

Also add the IMSIs of your devices under ``imsis``, for example::

    device-groups:
    - name:  "4g-oaisim-user"
      imsis:
        - "315010999912301"
        - "315010999912302"
        - "315010999912303"

Install AiaB as follows (specifying ``DATA_IFACE`` from above)::

    ENABLE_OAISIM=false DATA_IFACE=<iface> CHARTS=latest make 4g-core

Verifying the AiaB installation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Installing AiaB will take about 20 minutes with a fast Internet
connection.  If you see any errors / timeouts, try running the ``make``
command again.  The build will finish with a message:
“Your MME IP address is… ”  This is just the IP address assigned to
the ``DATA_IFACE``.   Remember this for the eNodeB setup.

When the install is complete, check that the 4G SD-CORE is running
as follows::

    $ kubectl -n omec get pod
    NAME                     READY   STATUS    RESTARTS   AGE
    cassandra-0              1/1     Running   0          7m27s
    config4g-0               1/1     Running   0          7m27s
    hss-0                    1/1     Running   0          7m27s
    mme-0                    4/4     Running   0          7m27s
    pcrf-0                   1/1     Running   0          7m27s
    simapp-65dc44b9d-stx6q   1/1     Running   0          7m27s
    spgwc-0                  2/2     Running   0          7m27s
    upf-0                    5/5     Running   0          7m27s

You should see all pods in Running status.

If you have installed the ROC, check that all its pods are running
as follows::

    $ kubectl -n aether-roc get pod
    NAME                                           READY   STATUS    RESTARTS   AGE
    aether-roc-api-78cc548bb9-7vjs2                1/1     Running   0          4m16s
    aether-roc-gui-v2-6d674fd446-tttb5             1/1     Running   0          4m16s
    aether-roc-umbrella-grafana-74f8489c8f-s9p45   2/2     Running   0          4m16s
    aether-roc-websocket-855d64549b-44fnc          1/1     Running   0          4m16s
    onos-cli-5d448ff6c4-stq5t                      1/1     Running   0          4m16s
    onos-config-7f4df96b88-vtp5s                   6/6     Running   0          4m16s
    onos-consensus-store-0                         1/1     Running   0          4m15s
    onos-topo-585c7c8976-6jq7b                     3/3     Running   0          4m16s
    sdcore-adapter-v2-5646d455b9-2d6zl             1/1     Running   0          4m15s

You should see all pods in Running status.

The UPF pod connects to the ``DATA_IFACE`` specified above using macvlan networks called
``core`` and ``access``.  Next, check that these have been successfully created, e.g. using
``ifconfig``::

    $ ifconfig core
    core: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.250.1  netmask 255.255.255.0  broadcast 192.168.250.255
        ether 16:9d:c1:0f:19:3a  txqueuelen 1000  (Ethernet)
        RX packets 513797  bytes 48400525 (48.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 102996  bytes 26530538 (26.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    $ ifconfig access
    access: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.252.1  netmask 255.255.255.0  broadcast 192.168.252.255
        ether 7a:9f:38:c0:18:15  txqueuelen 1000  (Ethernet)
        RX packets 558162  bytes 64064410 (64.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99553  bytes 16646682 (16.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Sercomm eNodeB setup
--------------------

The instructions in this section describe a basic configuration of
the eNodeB.  For a more comprehensive guide to
eNodeB configuration see `eNB Installation <https://docs.aetherproject.org/master/edge_deployment/enb_installation.html>`_.

The Sercomm eNodeB has two Ethernet ports: WAN and LAN.  We will use
the LAN port for configuration of the eNodeB and the WAN port for
normal operation.  Connect the eNodeB WAN port to the local network.

Connect the eNodeB LAN port to a free Ethernet port on a Linux machine
(say, a laptop) that will be used for the initial configuration of
the eNodeB.  On that machine run ``dhclient`` on the interface corresponding
to the Ethernet port, for example::

    sudo dhclient eth1

The interface should receive an IP address from the Sercomm eNodeB on
the 11.11.11.0/24 subnet.  Check this using ``ifconfig``::

    $ ifconfig eth1
    eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 11.11.11.100  netmask 255.255.255.0  broadcast 11.11.11.255
        inet6 fe80::2e0:4cff:fe68:2f76  prefixlen 64  scopeid 0x20<link>
        ether 00:e0:4c:68:2f:76  txqueuelen 1000  (Ethernet)
        RX packets 264652  bytes 216094312 (216.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183978  bytes 36528580 (36.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The eNodeB IP address should be 11.11.11.188 on that link.  Verify
this using ping::

    $ ping 11.11.11.188

To access the Sercomm eNodeB admin page, you can run a Web browser on
the laptop and direct it to `https://11.11.11.188`. Login to the admin
interface as user: ``sc_femto`` and password: ``scHt3pp``.

Click the ``NetWork set`` menu at the top. In the first section
``IP Address``, set *Connect type: DHCP* (assuming this is supported on
your local network, otherwise set up a static IP address).
Click the blue ``Save`` button at the bottom.

Next, click the ``Manage`` menu at the top and then click the
``LTE Basic Setting`` tab. Change these parameters and click ``Save``:

* Carrier Number: 2
* Carrier Aggregation: Unset
* BandWidth: 20
* FrequencyBand: 48,48
* EARFCN: 55440,55640
* CellIdentity: 2,1
* PCI: 100,101
* TxPower: 20
* Tunnel Type: IPv4
* MME IP Address: <MME IP address from AiaB installation>
* PLMNID: 315010
* TAC: 1
* Sync Source: FREE_RUNNING
* Enable CWMP: Unset

Click the ``SAS Configuration`` tab. In the ``Location Configuration``
section, enter these values and click “Save”:

* Location: Indoor
* Location Source: Manual
* Latitude: 0
* Longitude: 0
* Elevation: -18000

Next we need to add a static route to the UPF address, 192.168.252.3,
on the eNodeB.  Click on ``TR098`` menu and then click on ``IP`` tab.
Scroll down to ``Device.Routing.Router.`` section.  Click ``View List``.
Add this info on the blank line:

* Enable: Set
* StaticRoute: Set
* DestIPAddress: 192.168.252.0
* DestSubnetMask: 255.255.255.0
* GatewayIPAddress: <Use MME IP address from AiaB installation>
* Interface: Device.IP.Interface.1.

Then click the ``Add`` button at the far right.

Finally click the ``FAPService`` menu and then go to the ``FAPControl``
tab.  Check the box next to ``AdminState`` in the first section and
click ``Save``.

After these changes are made, reboot the eNodeB by clicking the red
``power button`` square at top right and selecting ``Reboot``.  When the
eNodeB comes back up, it should have an IP address on the network
(via the WAN port), and the admin page should now be available on
`https://<endoeb-ip>`.

Test connectivity from the eNodeB to the MME and the UPF running on
the server as follows.  Login to the eNodeB admin interface, click
the “Manage” menu at the top, and click the ``IP Diagnose`` tab.  Under
``Ping and Traceroute``, select ``ping``, and then type the following IP
addresses into the box to the right and click ``Run``:

* <MME IP address from AiaB installation>
* 192.168.251.1
* 192.168.252.3

If all of these are working, then you are ready to try to connect
devices to the network.

Connecting Devices
------------------

Documenting how to configure different types of devices to work
with Aether is work-in-progress.

Enable Monitoring
-----------------

To easily monitor your AiaB deployment, you can run the following command to
deploy the Prometheus/Grafana monitoring stack on your AiaB server::

    make monitoring-4g

After this step, Grafana is available at `http://<server-ip>:30950`.  You will see a number of system dashboards
for monitoring Kubernetes, as well as a simple AiaB dashboard that enables inspection of the local Aether state.

.. figure:: images/4g-aiab-grafana-dashboard.png
    :align: center
    :width: 100 %

    *4G Grafana AiaB Dashboard*

The dashboard shows whether the eNodeB is connected to the core, how many active UEs there are, and the
uplink (Tx Bitrate) and downlink (Rx Bitrate) throughput at the UPF.

To create a new dashboard for Aether, you can login to Grafana as user `admin` with password `prom-operator`.
You can also modify an existing dashboard by making a copy of the dashboard and editing that.  Note that any
dashboard changes will be lost if Grafana is restarted; if you want to make your dashboard more permanent, you can
save its JSON to a file, add the file to directory `aether-in-a-box/resources/4g-monitoring`, and edit
`kustomization.yaml` in that directory to add the file to the ConfigMapGenerator block.  After these
steps, re-running `make monitoring-4g` should install the dashboard in a ConfigMap and Grafana should
automatically load it.

Troubleshooting
---------------

AiaB connects macvlan networks to ``DATA_IFACE`` so that the UPF can communicate on the network.
To do this it assumes that the *systemd-networkd* service is installed and running, ``DATA_IFACE``
is under its control, and the systemd-networkd configuration file for ``DATA_IFACE`` ends with
``<DATA_IFACE>.network``, where ``<DATA_IFACE>`` stands for the actual interface name.  It
tries to find this configuration file by looking in the standard paths.  If it fails you'll see
a message like::

    FATAL: Could not find systemd-networkd config for interface foobar, exiting now!
    make: *** [Makefile:112: /users/acb/aether-in-a-box//build/milestones/interface-check] Error 1

In this case, you can specify a ``DATA_IFACE_PATH=<path to the config file>`` argument to ``make``
so that AiaB can find the systemd-networkd configuration file for ``DATA_IFACE``.

Restarting the AiaB Server
--------------------------

AiaB should come up in a mostly working state if the AiaB server is rebooted.  If any pods are
stuck in an Error or CrashLoopBackoff state they can be restarted using ``kubectl delete pod``.
It might also be necessary to power cycle the Sercomm eNodeB in order to get it to reconnect to
the SD-CORE.