blob: 7b3e7003b917de93cd0673a1728e61b1e92ef9db [file] [log] [blame]
..
SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
SPDX-License-Identifier: Apache-2.0
Runtime Deployment
==================
This section describes how to install and configure Aether Edge Runtime including Kubernetes
and system level applications listed below.
* sealed-secrets
* rancher-monitoring
* fluent-bit
* opendistro-es
* hostpath-provisioner
* edge-maintenance-agent
* sriov-device-plugin
* uedns
For this, we will be using GitOps based CI/CD systems and what you will need to do is
create patches in Aether GitOps repositories, **aether-pod-configs** and **aether-app-configs**,
to provide the cluster configurations to the CI/CD systems.
.. attention::
If you skipped VPN bootstap step and didn't add the deployment jobs for the new edge,
go to :ref:`Add deployment jobs <add_deployment_jobs>` step and finish it first
before proceeding.
K8S cluster deployment
----------------------
Download ``aether-pod-configs`` repository if you don't have it already in
your development machine.
.. code-block:: shell
$ cd $WORKDIR
$ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs"
.. attention::
If you skipped VPN bootstap step and didn't update global resource maps for the new edge,
go to :ref:`Update global resource maps <update_global_resource>` step and
finish ``cluster_map.tfvars`` and ``user_map.tfvars`` update first before proceeding.
Run the following commands to automatically generate Terraform configurations needed to
create a new cluster in `Rancher <https://rancher.aetherproject.org>`_ and add the servers
and switches to the cluster.
.. code-block:: shell
# Create ace_cofig.yaml file if you haven't yet
$ cd $WORKDIR/aether-pod-configs/tools
$ cp ace_config.yaml.example ace_config.yaml
$ vi ace_config.yaml
# Set all values
$ make runtime
Created ../production/ace-test/provider.tf
Created ../production/ace-test/cluster.tf
Created ../production/ace-test/rke-bare-metal.tf
Created ../production/ace-test/addon-manifests.yml.tpl
Created ../production/ace-test/project.tf
Created ../production/ace-test/member.tf
Created ../production/ace-test/backend.tf
Created ../production/ace-test/cluster_val.tfvars
Create a review request.
.. code-block:: shell
$ cd $WORKDIR/aether-pod-configs
$ git add .
$ git commit -m "Add test ACE runtime configs"
$ git review
Once your review request is accepted and merged, Aether CI/CD system starts to deploy K8S.
Wait until the cluster status changes to **Active** in `Rancher <https://rancher.aetherproject.org>`_.
It normally takes 10 - 15 minutes depending on the speed of the container images
download at the edge.
System Application Deployment
-----------------------------
For the system application deployment, we will be using Rancher's built-in GitOps tool, **Fleet**.
Fleet uses a git repository as a single source of truth to manage applications in the clusters.
For Aether, **aether-app-configs** is the repository where all Aether applications
are defined.
Most of the Aether system applications do not require cluster specific configurations,
except **rancher-monitoring** and **uedns**.
For these applications, you will have to manually create custom configurations and
commit them to aether-app-configs.
First, download ``aether-app-configs`` if you don't have it already in your development machine.
.. code-block:: shell
$ cd $WORKDIR
$ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-app-configs"
Configure rancher-monitoring
############################
Open ``fleet.yaml`` under ``infrastructure/rancher-monitoring``, add a custom target
with the new cluster name as a selector, and provide cluster specific Helm values and
kustomize overlay directory path like below.
.. code-block:: yaml
$ cd $WORKDIR/aether-app-configs/infrastructure/rancher-monitoring
$ vi fleet.yaml
# add following block at the end
- name: ace-test
clusterSelector:
matchLabels:
management.cattle.io/cluster-display-name: ace-test
helm:
values:
prometheus:
prometheusSpec:
additionalAlertRelabelConfigs:
- source_labels: [__address__]
target_label: cluster
replacement: ace-test
kustomize:
dir: overlays/prd-ace
.. note::
Above step will not be required in Rancher v2.6 as it supports using cluster labels as helm values in a list.
Configure ue-dns
################
For UE-DNS, it is required to create a Helm values file for the new cluster.
You'll need cluster domain and kube-dns ClusterIP. Both can be found in
``aether-pod-configs/production/cluster_map.tfvars``.
Be sure to replace ``[ ]`` in the example configuration below to the actual cluster values.
.. code-block:: yaml
$ cd $WORKDIR/aether-app-configs/infrastructure/coredns
$ mkdir overlays/prd-ace-test
$ vi overlays/prd-ace-test/values.yaml
# SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org>
serviceType: ClusterIP
service:
clusterIP: [next address of the kube-dns ip]
servers:
- zones:
- zone: .
port: 53
plugins:
- name: errors
- name: health
configBlock: |-
lameduck 5s
- name: ready
- name: prometheus
parameters: 0.0.0.0:9153
- name: forward
parameters: . /etc/resolv.conf
- name: cache
parameters: 30
- name: loop
- name: reload
- name: loadbalance
- zones:
- zone: aetherproject.net
port: 53
plugins:
- name: errors
- name: rewrite continue
configBlock: |-
name regex (.*)\.aetherproject.net {1}.svc.[cluster domain]
answer name (.*)\.svc\.[cluster domain] {1}.aetherproject.net
- name: forward
parameters: . [kube-dns ip]
configBlock: |-
except kube-system.svc.[cluster domain] aether-sdcore.svc.[cluster domain] tost.svc.[cluster domain]
- name: cache
parameters: 30
Next, update ``fleet.yaml`` under ``infrastructure/coredns`` so that Fleet can use the custom configuration
you just created when deploying UE-DNS to the cluster.
.. code-block:: yaml
$ cd $WORKDIR/aether-app-configs/infrastructure/coredns
$ vi fleet.yaml
# add following block at the end
- name: prd-ace-test
clusterSelector:
matchLabels:
management.cattle.io/cluster-display-name: ace-test
helm:
valuesFiles:
- overlays/prd-ace-test/values.yaml
Submit your changes.
.. code-block:: shell
$ cd $WORKDIR/aether-app-configs
$ git status
$ git add .
$ git commit -m "Add test ACE application configs"
$ git review
Assign Fleet workspace
######################
By default, all new clusters are assgiend to a default Fleet workspace called **fleet-default**.
To make a cluster part of Aether and have the applications defined in aether-app-configs deployed,
you must assign the cluster to either **aether-stable** or **aether-alpha** workspace.
For clusters expecting minimal downtime, assign to **aether-stable**.
For clusters for development or previewing upcoming release, assign to **aether-alpha**.
Log in to `Rancher <https://rancher.aetherproject.org>`_ as ``admin`` or ``onfadmin`` user
and go to the **Cluster Explorer**.
In the top left dropdown menu, click **Cluster Explorer > Continuous Delivery**.
.. image:: images/fleet-move-workspace.png
1) Click the second dropdown menu from the left at the top and select **fleet-default**.
2) Select **Clusters** on the left menu and you'll see the new cluster.
3) Click the checkbox in front of the cluster name.
4) Select **Assign to...** button and assign the cluster to the Aether workspace.
Switch the workspace to the Aether workspace, click **Clusters** in the left menu, and check the
new cluster exists.
Wait until the cluster state becomes **Active**.
.. attention::
Ignore BESS UPF failure at this point.