Updating UPF deployment mode with DPDK
Signed-off-by: Marikkannu, Suresh <suresh.marikkannu@intel.com>
Change-Id: I436e5e5dbba755333043db3683c9f40b196c862e
diff --git a/deployment/deploymentupf.rst b/deployment/deploymentupf.rst
index 8c4931c..47b163a 100644
--- a/deployment/deploymentupf.rst
+++ b/deployment/deploymentupf.rst
@@ -19,7 +19,7 @@
- UPF Attach/detach to SD-Core. UPF Pools created based on enterprise need.
Option1: AF_PACKET Mode UPF
-''''''''''''''''''''''''''''
+----------------------------
- Refer Aether in a box (AIAB) for this mode.
- AIAB has all default values and its good example of how to use AF_PACKET mode
@@ -28,7 +28,203 @@
Option2: SRIOV and DPDK enabled UPF
-'''''''''''''''''''''''''''''''''''''
+------------------------------------
+Please follow the below procedure to bring up UPF in DPDK mode with SRIOV.
-- This option should be preferred if performance is utmost important
-- Please refer to `UPF Installation Guide <https://docs.google.com/document/d/1-BT7XqVsL7ffBlD7aweYaScKDQH7Gv5tHKt-sJGuf6c/edit#>`_ guide for more details
+Pre-requisite:
+''''''''''''''
+- As a pre-requisite please make sure virtualization and VT-d parameters are enabled in BIOS.
+
+- make sure enough hugepage memory allocated, iommu enabled. These changes can be made by updating
+ below parameter in /etc/default/grub as follows,
+
+ .. code-block::
+
+ GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=32 transparent_hugepage=never"
+
+ Note: Number of hugepages = 2 X No of UPF Instances
+
+ Once it is updated apply the changes by running below command,
+
+ .. code-block::
+
+ $sudo update-grub
+ $sudo reboot
+
+ You can verify the allocated hugepages using below command,
+
+ .. code-block::
+
+ curl https://raw.githubusercontent.com/DPDK/dpdk/main/usertools/dpdk-hugepages.py -O dpdk-hugepages.py
+
+ $chmod +x dpdk-hugepages.py
+ $./dpdk-hugepages.py -s
+ Node Pages Size Total
+ 0 2048 2Mb 4Gb
+ 1 2048 2Mb 4Gb
+
+ Hugepages mounted on /dev/hugepages
+
+step 1: Create VF devices and bind them to the vfio-pci driver
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+- Create required VF devices as follows (In this example the PF interface used is "ens801f0")
+
+ .. code-block::
+
+ echo 2 > /sys/class/net/ens801f0/device/sriov_numvfs
+
+ Now retrieve the PCI address for the newly created VF devices using below command,
+
+ .. code-block::
+
+ ls -l /sys/class/net/ens801f0/device/virtfn*
+
+- Bind the VF devices to the vfio-pci driver as follows,
+
+ .. code-block::
+
+ curl https://raw.githubusercontent.com/ceph/dpdk/master/tools/dpdk-devbind.py -O dpdk-devbind.py
+ chmod +x dpdk-devbind.py
+
+ ./dpdk-devbind.py -b vfio-pci 0000:b1:01.0
+ ./dpdk-devbind.py -b vfio-pci 0000:b1:01.1
+
+
+step 2 - Install SRIOV device plugin
+''''''''''''''''''''''''''''''''''''
+- Install the required packages for kubernetes,
+
+ .. code-block::
+
+ cd aether-in-a-box
+ make node-prep
+
+- Download sriov-device-plugin.yaml and install,
+
+ .. code-block::
+
+ $wget https://github.com/opennetworkinglab/aether-configs/blob/main/sys/sriov-device-plugin/sriov-device-plugin.yaml
+ $kubectl apply -f sriov-device-plugin.yaml
+
+- Create the sriov-device-plugin-config.yaml file with below details and install,
+
+ .. code-block::
+
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+ name: sriovdp-config
+ data:
+ config.json: |
+ {
+ "resourceList": [
+ {
+ "resourcePrefix": "intel.com",
+ "resourceName": "intel_sriov_vfio_access",
+ "selectors": {
+ "pfNames": ["ens801f0#0-1"],
+ "vendors": ["8086"],
+ "drivers": ["vfio-pci"]
+ }
+ },
+ {
+ "resourcePrefix": "intel.com",
+ "resourceName": "intel_sriov_vfio_core",
+ "selectors": {
+ "pfNames": ["ens801f0#2-3"],
+ "vendors": ["8086"],
+ "drivers": ["vfio-pci"]
+ }
+ }
+ ]
+ }
+
+ $kubectl apply -f sriov-device-plugin-config.yaml
+
+- Make sure that there are minimum 1 intel_sriov_vfio_access/intel_sriov_vfio_core resources available,
+
+ .. code-block::
+
+ $kubectl get nodes -o json | jq '.items[].status.allocatable'
+ {
+ "cpu": "144",
+ "ephemeral-storage": "222337451653",
+ "hugepages-1Gi": "32Gi",
+ "intel.com/intel_sriov_vfio_access": "1",
+ "intel.com/intel_sriov_vfio_core": "1",
+ "memory": "494544488Ki",
+ "pods": "110"
+ }
+
+step 3 - Deploy 5G core using AiaB
+'''''''''''''''''''''''''''''''''''
+
+Update sd-core-5g-values.yaml file parameters as follows (along with any other changes
+required with respect to the environment),
+
+ .. code-block::
+
+ diff --git a/sd-core-5g-values.yaml b/sd-core-5g-values.yaml
+ index 58232ad..1c8893d 100644
+ --- a/sd-core-5g-values.yaml
+ +++ b/sd-core-5g-values.yaml
+ @@ -224,7 +224,7 @@ omec-sub-provision:
+ omec-user-plane:
+ enable: true
+ resources:
+ - enabled: false
+ + enabled: true
+ images:
+ repository: "registry.opennetworking.org/docker.io/"
+ # uncomment below section to add update bess image tag
+ @@ -234,12 +234,13 @@ omec-user-plane:
+ config:
+ upf:
+ name: "oaisim"
+ + privileged: true
+ sriov:
+ - enabled: false #default sriov is disabled in AIAB setup
+ + enabled: true #default sriov is disabled in AIAB setup
+ hugepage:
+ - enabled: false #should be enabled if dpdk is enabled
+ + enabled: true #should be enabled if dpdk is enabled
+ #can be any other plugin as well, remember this plugin dictates how IP address are assigned.
+ - cniPlugin: macvlan
+ + cniPlugin: vfioveth
+ ipam: static
+ routes:
+ - to: ${NODE_IP}
+ @@ -247,12 +248,16 @@ omec-user-plane:
+ enb:
+ subnet: ${RAN_SUBNET} #this is your gNB network
+ access:
+ - iface: ${DATA_IFACE}
+ + resourceName: "intel.com/intel_sriov_vfio_access"
+ + ip: "192.168.252.3/24"
+ + gateway: "192.168.252.1"
+ core:
+ - iface: ${DATA_IFACE}
+ + resourceName: "intel.com/intel_sriov_vfio_core"
+ + ip: "192.168.250.3/24"
+ + gateway: "192.168.250.1"
+ cfgFiles:
+ upf.json:
+ - mode: af_packet #this mode means no dpdk
+ + mode: dpdk #this mode means no dpdk
+
+- Deploy the 5g-core (in the below case GNBSIM is disabled) as required,
+
+ .. code-block::
+
+ ENABLE_GNBSIM=false DATA_IFACE=ens801f0 CHARTS=latest make 5g-core
+
+ UPF will be deployed with DPDK now and you can verify the traffic using UERANSIM (or any preferred method). If you want to deploy the Aether with RoC then use below command,
+
+ .. code-block::
+
+ ENABLE_GNBSIM=false DATA_IFACE=ens801f0 CHARTS=latest make roc-5g-models 5g-core
+
+.. note::
+
+ - This option should be preferred if performance is utmost important
+ - Please refer to `UPF Installation Guide <https://docs.google.com/document/d/1-BT7XqVsL7ffBlD7aweYaScKDQH7Gv5tHKt-sJGuf6c/edit#>`_ guide for more details
diff --git a/developer/aiab.rst b/developer/aiab.rst
index 82e6018..1080a7c 100644
--- a/developer/aiab.rst
+++ b/developer/aiab.rst
@@ -7,24 +7,5 @@
Aether In a Box - 4G
====================
-The documentation for Aether In a Box (AiaB) - 4G is located in `Installing the 4G AIAB <https://docs.aetherproject.org/master/developer/aiab.html#installing-the-4g-aiab>`_,
-where specifics about setting up AiaB 4G, running 4G test, troubleshooting,
-and other details are described. Moreover, details about deploying AiaB
-with external 4G radios are presented `Aether-in-a-Box with External 4G Radio <https://docs.aetherproject.org/master/developer/aiabhw.html>`_.
-
-Developer Loop
-______________
-
-Suppose you wish to test a new build of a 4G SD-CORE services. You can deploy
-custom images by editing ~/aether-in-a-box/sd-core-4g-values.yaml, for
-example::
-
- images:
- tags:
- spgwc: omecproject/spgw:master-e419062
-
-To upgrade a running 4G SD-CORE with the new image, or to redeploy the 4G
-SD-CORE with the image::
-
- make reset-test # delete 4G deployment if it was already started before updating image
- make 4g-test # now this deployment will use new spgwc image
+Currently `Aether OnRamp <https://docs.aetherproject.org/master/onramp/overview.html>`_
+doesn't support 4G deployment.
diff --git a/developer/aiab5g.rst b/developer/aiab5g.rst
index 4e19543..3c8f3c8 100644
--- a/developer/aiab5g.rst
+++ b/developer/aiab5g.rst
@@ -7,10 +7,9 @@
Aether In a Box - 5G
====================
-The documentation for Aether In a Box (AiaB) - 5G is located in `Installing the 5G AIAB <https://docs.aetherproject.org/master/developer/aiab.html#installing-the-5g-aiab>`_,
+The documentation for Aether In a Box (AiaB) - 5G is located in `Aether OnRamp <https://docs.aetherproject.org/master/onramp/overview.html>`_,
where specifics about setting up AiaB 5G, running 5G test, troubleshooting,
-and other details are described. Moreover, details about deploying AiaB
-with external 5G radios are presented `Aether-in-a-Box with External 5G Radio <https://docs.aetherproject.org/master/developer/aiabhw5g.html>`_.
+and other details are described.
Developer Loop
______________
diff --git a/dict.txt b/dict.txt
index bca1952..1e7ba1c 100644
--- a/dict.txt
+++ b/dict.txt
@@ -143,12 +143,16 @@
hss
hssdb
http
+hugepage
+hugepages
hyperscale
hyperscaler
iUPF
imsi
imsis
instantiation
+intel
+iommu
iot
ip
iptables
@@ -199,6 +203,7 @@
pcap
pcaps
pcf
+pci
pcrf
pcrfdb
pdusessest
@@ -256,6 +261,7 @@
upfadapter
uplink
verifiably
+vfio
virtualenv
webconsole
webhooks