Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 1 | .. |
| 2 | SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org> |
| 3 | SPDX-License-Identifier: Apache-2.0 |
| 4 | |
| 5 | BESS UPF Deployment |
| 6 | =================== |
| 7 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 8 | This section describes how to configure and deploy BESS UPF. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 9 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 10 | Network Configuration |
| 11 | --------------------- |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 12 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 13 | BESS UPF enabled edge setup requires three additional user plane subnets |
| 14 | apart from the default K8S subnets. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 15 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 16 | * **enb**: Used to provide eNBs with connectivity to SD-Core and UPF. |
| 17 | * **access**: Used to provide UPF with connectivity to eNBs. |
| 18 | * **core**: Used to provide UPF with edge services as well as the Internet access. |
| 19 | |
| 20 | To help your understanding, the following example ACE environment will be used in the rest of the guide. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 21 | |
Zack Williams | 1ae109e | 2021-07-27 11:17:04 -0700 | [diff] [blame] | 22 | .. image:: images/bess-upf-example-network.svg |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 23 | |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 24 | .. note:: |
| 25 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 26 | Admin and out-of-band networks are not depicted in the diagram. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 27 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 28 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 29 | | Network | VLAN ID | Subnet | Interface | IP address | |
| 30 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 31 | | k8smgmt | 1 | 192.168.1.0/24 (gw: 192.168.1.1) | management server | 192.168.1.254 | |
| 32 | | | | +-------------------+---------------+ |
| 33 | | | | | compute1 | 192.168.1.3 | |
| 34 | | | | +-------------------+---------------+ |
| 35 | | | | | compute2 | 192.168.1.4 | |
| 36 | | | | +-------------------+---------------+ |
| 37 | | | | | compute3 | 192.168.1.5 | |
| 38 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 39 | | enb | 2 | 192.168.2.0/24 (gw: 192.168.2.1) | enb1 | 192.168.2.10 | |
| 40 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 41 | | access | 3 | 192.168.3.0/24 (gw: 192.168.3.1) | upf1 access | 192.168.3.10 | |
| 42 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
| 43 | | core | 4 | 192.168.4.0/24 (gw: 192.168.4.1) | management server | 192.168.4.254 | |
| 44 | | | | +-------------------+---------------+ |
| 45 | | | | | upf1 core | 192.168.4.10 | |
| 46 | +-----------+-----------+------------------------------------+-------------------+---------------+ |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 47 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 48 | It is assumed that the management server has the only external routable address and acts |
| 49 | as a router connecting the Aether pod to the outside. |
| 50 | This means that all uplink packets leaving the Aether pod needs to be masqueraded with the |
| 51 | external address of the management server or the k8smgmt address if the destination |
| 52 | is Aether central. |
| 53 | Also, in order for downlink traffic to UE to be delivered to its destination, |
| 54 | it must be forwarded to the UPF's core interface. |
| 55 | This adds additional routes to the management server and L3 switch. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 56 | |
| 57 | |
| 58 | Check Cluster Resources |
| 59 | ----------------------- |
| 60 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 61 | Before proceeding with the deployment, make sure the cluster has enough resources |
| 62 | to run BESS UPF by running the command below. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 63 | |
| 64 | .. code-block:: shell |
| 65 | |
Hyunsun Moon | 53a18a7 | 2021-08-10 16:57:32 -0700 | [diff] [blame] | 66 | $ kubectl get nodes -o json | jq '.items[].status.allocatable' |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 67 | { |
| 68 | "cpu": "95", |
| 69 | "ephemeral-storage": "1770223432846", |
| 70 | "hugepages-1Gi": "32Gi", |
| 71 | "intel.com/intel_sriov_netdevice": "32", |
| 72 | "intel.com/intel_sriov_vfio": "32", |
| 73 | "memory": "360749956Ki", |
| 74 | "pods": "110" |
| 75 | } |
| 76 | |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 77 | For best performance, BESS UPF requires the following resources: |
| 78 | |
| 79 | * 2 dedicated cores (``"cpu"``) |
| 80 | * 2 1GiB HugePages (``"hugepages-1Gi"``) |
| 81 | * 2 SRIOV Virtual Functions bound to **vfio-pci** driver (``"intel.com/intel_sriov_vfio"``) |
| 82 | |
| 83 | For environments where these resources are not available, contact Ops team for |
| 84 | advanced configuration. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 85 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 86 | Configure and Deploy |
| 87 | -------------------- |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 88 | |
| 89 | Download ``aether-app-configs`` if you don't have it already in your development machine. |
| 90 | |
| 91 | .. code-block:: shell |
| 92 | |
| 93 | $ cd $WORKDIR |
| 94 | $ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-app-configs" |
| 95 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 96 | Move the directory to ``apps/bess-upf/upf1`` and create a Helm values file for the new cluster as shown below. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 97 | Don't forget to replace the IP addresses in the example configuration with the addresses of the actual cluster. |
| 98 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 99 | .. code-block:: yaml |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 100 | |
| 101 | $ cd $WORKDIR/aether-app-configs/apps/bess-upf/upf1 |
| 102 | $ mkdir overlays/prd-ace-test |
| 103 | $ vi overlays/prd-ace-test/values.yaml |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 104 | # SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org> |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 105 | |
| 106 | config: |
| 107 | upf: |
| 108 | enb: |
| 109 | subnet: "192.168.2.0/24" |
| 110 | access: |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 111 | ip: "192.168.3.10/24" |
| 112 | gateway: "192.168.3.1" |
Hyunsun Moon | 6001f94 | 2021-08-01 10:35:04 -0700 | [diff] [blame] | 113 | vlan: 3 |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 114 | core: |
Hyunsun Moon | 015eb21 | 2021-08-24 18:06:24 -0700 | [diff] [blame] | 115 | ip: "192.168.4.10/24" |
| 116 | gateway: "192.168.4.1" |
Hyunsun Moon | 6001f94 | 2021-08-01 10:35:04 -0700 | [diff] [blame] | 117 | vlan: 4 |
Hyunsun Moon | ba65df2 | 2021-08-10 20:01:06 -0700 | [diff] [blame] | 118 | # Add below when connecting to 5G core |
| 119 | #cfgFiles: |
| 120 | # upf.json: |
| 121 | # cpiface: |
| 122 | # dnn: "8internet" |
| 123 | # hostname: "upf" |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 124 | |
| 125 | |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 126 | Update ``fleet.yaml`` in the same directory to let Fleet use the custom configuration when deploying |
| 127 | BESS UPF to the new cluster. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 128 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 129 | .. code-block:: yaml |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 130 | |
| 131 | $ vi fleet.yaml |
| 132 | # add following block at the end |
| 133 | - name: prd-ace-test |
| 134 | clusterSelector: |
| 135 | matchLabels: |
| 136 | management.cattle.io/cluster-display-name: ace-test |
| 137 | helm: |
| 138 | valuesFiles: |
| 139 | - overlays/prd-ace-test/values.yaml |
| 140 | |
| 141 | |
Hyunsun Moon | 2b2bf9a | 2021-08-01 05:29:48 -0700 | [diff] [blame] | 142 | Submit your changes. |
Hyunsun Moon | d12f32b | 2021-07-30 12:55:00 -0700 | [diff] [blame] | 143 | |
| 144 | .. code-block:: shell |
| 145 | |
| 146 | $ cd $WORKDIR/aether-app-configs |
| 147 | $ git status |
| 148 | $ git add . |
| 149 | $ git commit -m "Add BESS UPF configs for test ACE" |
| 150 | $ git review |
| 151 | |
| 152 | |
| 153 | Go to Fleet dashboard and wait until the cluster status becomes **Active**. |
Hyunsun Moon | b9c7dde | 2021-08-01 06:11:08 -0700 | [diff] [blame] | 154 | It can take up to 1 min for Fleet to fetch the configuration updates. |