Merge "Update vpn and runtime deployment guides"
diff --git a/amp/monitoring.rst b/amp/monitoring.rst
new file mode 100644
index 0000000..b9d2ff4
--- /dev/null
+++ b/amp/monitoring.rst
@@ -0,0 +1,91 @@
+..
+ SPDX-FileCopyrightText: © 2021 Open Networking Foundation <support@opennetworking.org>
+ SPDX-License-Identifier: Apache-2.0
+
+Monitoring and Alerts
+=====================
+
+Aether leverages `Prometheus <https://prometheus.io/docs/introduction/overview/>`_ to collect
+and store platform and service metrics, `Grafana <https://grafana.com/docs/grafana/latest/getting-started/>`_
+to visualize metrics over time, and `Alertmanager <https://prometheus.io/docs/alerting/latest/alertmanager/>`_ to
+notify Aether OPs staff of events requiring attention. This monitoring stack is running on each Aether cluster.
+This section describes how an Aether component can "opt in" to the Aether monitoring stack so that its metrics can be
+collected and graphed, and can trigger alerts.
+
+
+Exporting Service Metrics to Prometheus
+---------------------------------------
+An Aether component implements a `Prometheus exporter <https://prometheus.io/docs/instrumenting/writing_exporters/>`_
+to expose its metrics to Prometheus. An exporter provides the current values of a components's
+metrics via HTTP using a simple text format. Prometheus scrapes the exporter's HTTP endpoint and stores the metrics
+in its Time Series Database (TSDB) for querying and analysis. Many `client libraries <https://prometheus.io/docs/instrumenting/clientlibs/>`_
+are available for instrumenting code to export metrics in Prometheus format. If a component's metrics are available
+in some other format, tools like `Telegraf <https://docs.influxdata.com/telegraf>`_ can be used to convert the metrics
+into Prometheus format and export them.
+
+A component that exposes a Prometheus exporter HTTP endpoint via a Service can tell Prometheus to scrape
+this endpoint by defining a
+`ServiceMonitor <https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md>`_
+custom resource. The ServiceMonitor is typically created by the Helm chart that installs the component.
+
+
+Working with Grafana Dashboards
+--------------------------------
+Once the local cluster's Prometheus is collecting a component's metrics, they can be visualized using Grafana
+dashboards. The Grafana instance running on the AMP cluster is able to send queries to the Prometheus
+servers running on all Aether clusters. This means that component metrics can be visualized on the AMP Grafana
+regardless of where the component is actually running.
+
+In order to create a new Grafana dashboard or modify an existing one, first login to the AMP Grafana using an account
+with admin privileges. To add a new dashboard, click the **+** at left. To make a copy of an existing dashboard for
+editing, click the **Dashboard Settings** icon (gear icon) at upper right of the existing dashboard, and then
+click the **Save as…** button at left.
+
+Next, add panels to the dashboard. Since Grafana can access Prometheus on all the clusters in the environment,
+each cluster is available as a data source. For example, when adding a panel showing metrics collected on the
+ace-menlo cluster, choose ace-menlo as the data source.
+
+Clicking on the floppy disk icon at top will save the dashboard *temporarily* (the dashboard is not
+saved to persistent storage and is deleted as soon as Grafana is restarted). To save the dashboard *permanently*,
+click the **Share Dashboard** icon next to the title and save its JSON to a file. Then add the file to the
+aether-app-configs repository so that it will be deployed by Fleet:
+
+* Change to directory ``aether-app-configs/infrastructure/rancher-monitoring/overlays/<amp-cluster>/``
+* Copy the dashboard JSON file to the ``dashboards/`` sub-directory
+* Edit ``kustomization.yaml`` and add the new dashboard JSON under ``configmapGenerator``
+* Commit the changes and submit patchset to gerrit
+
+Once the patchset is merged, the AMP Grafana will automatically detect and deploy the new dashboard.
+
+Adding Service-specific Alerts
+------------------------------
+An alert can be triggered in Prometheus when a component metric crosses a threshold. The Alertmanager
+then routes the alert to one or more receivers (e.g., an email address or Slack channel).
+
+To add an alert for a component, create a
+`PrometheusRule <https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md>`_
+custom resource, for example in the Helm chart that deploys the component. This resource describes one or
+more `rules <https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/>`_ using Prometheus expressions;
+if the expression is true for the time indicated, then the alert is raised. Once the PrometheusRule
+resource is instantiated, the cluster's Prometheus will pick up the rule and start evaluating it.
+
+The Alertmanager is configured to send alerts with *critical* or *warning* severity to e-mail and Slack channels
+monitored by Aether OPs staff. If it is desirable to route a specific alert to a different receiver
+(e.g., a component-specific Slack channel), it is necessary to change the Alertmanager configuration. This is stored in
+a `SealedSecret <https://github.com/bitnami-labs/sealed-secrets>`_ custom resource in the aether-app-configs repository.
+To update the configuration:
+
+* Change to directory ``aether-app-configs/infrastructure/rancher-monitoring/overlays/<cluster>/``
+* Update the ``receivers`` and ``route`` sections of the ``alertmanager-config.yaml`` file
+* Encode the ``alertmanager-config.yaml`` file as a Base64 string
+* Create a file ``alertmanager-config-secret.yaml`` to define the Secret resource using the Base64-encoded string
+* Run the following command using a valid ``PUBLICKEY``:
+
+.. code-block:: shell
+
+ $ kubeseal --cert "${PUBLICKEY}" --scope cluster-wide --format yaml < alertmanager-config-secret.yaml > alertmanager-config-sealed-secret.yaml
+
+* Commit the changes and submit patchset to gerrit
+
+Once the patchset is merged, verify that the SealedSecret was successfully unsealed and converted to a Secret
+by looking at the logs of the *sealed-secrets-controller* pod running on the cluster in the *kube-system* namespace.
diff --git a/developer/images/rocgui.png b/developer/images/rocgui.png
new file mode 100644
index 0000000..c2b388f
--- /dev/null
+++ b/developer/images/rocgui.png
Binary files differ
diff --git a/developer/roc.rst b/developer/roc.rst
new file mode 100644
index 0000000..a3b2f9b
--- /dev/null
+++ b/developer/roc.rst
@@ -0,0 +1,231 @@
+.. vim: syntax=rst
+
+Aether ROC Developer Guide
+==========================
+
+Background / Development Environment
+------------------------------------
+
+This document assumes familiarity with Kubernetes and Helm, and that a Kubernetes/Helm development
+environment has already been deployed in the developer’s work environment.
+This development environment can use any of a number of potential mechanisms -- including KinD, Kubeadm, etc.
+The Aether-in-a-Box script is one potential way to setup a development environment, but not the only way.
+As an alternative to the developer’s local machine, a remote environment can be set up, for example on
+cloud infrastructure such as cloudlab.
+
+Installing Prerequisites
+------------------------
+
+Atomix and onos-operator must be installed::
+
+ # create necessary namespaces
+ kubectl create namespace micro-onos
+
+ # install atomix
+ helm -n kube-system install atomix-controller atomix/atomix-controller
+ helm -n kube-system install atomix-raft-storage atomix/atomix-raft-storage
+
+ # install the onos operator
+ helm install -n kube-system onos-operator onosproject/onos-operator
+
+
+Verify that these services were installed properly.
+You should see pods for *atomix-controller*, *atomix-raft-storage-controller*,
+*onos-operator-config*, and *onos-operator-topo*.
+Execute these commands::
+
+ kubectl -n kube-system get pods | grep -i atomix
+ kubectl -n kube-system get pods | grep -i onos
+
+
+Create a values-override.yaml
+-----------------------------
+
+You’ll want to override several of the defaults in the ROC helm charts::
+
+ cat > values-override.yaml <<EOF
+ import:
+ onos-gui:
+ enabled: true
+
+ onos-gui:
+ ingress:
+ enabled: false
+
+ sdcore-adapter-v3:
+ prometheusEnabled: false
+
+ sdcore-exporter:
+ prometheusEnabled: false
+
+ onos-exporter:
+ prometheusEnabled: false
+
+ aether-roc-gui-v3:
+ ingress:
+ enabled: false
+ EOF
+
+Installing the Aether-Roc-Umbrella Helm chart
+---------------------------------------------
+
+Add the necessary helm repositories::
+
+ # obtain username and password from Michelle and/or ONF infra team
+ export repo_user=<username>
+ export repo_password=<password>
+ helm repo add sdran --username "$repo_user" --password "$repo_password" https://sdrancharts.onosproject.org
+
+Aether-Roc-Umbrella will bring up the ROC and its services::
+
+ helm -n micro-onos install aether-roc-umbrella sdran/aether-roc-umbrella -f values-override.yaml
+
+ kubectl wait pod -n micro-onos --for=condition=Ready -l type=config --timeout=300s
+
+
+Posting the mega-patch
+----------------------
+
+The ROC usually comes up in a blank state -- there are no Enterprises, UEs, or other artifacts present in it.
+The mega-patch is an example patch that populates the ROC with some sample enterprises, UEs, slices, etc.
+Execute the following::
+
+ # launch a port-forward for the API
+ # this will continue to run in the background
+ kubectl -n micro-onos port-forward service/aether-roc-api --address 0.0.0.0 8181:8181 &
+
+ git clone https://github.com/onosproject/aether-roc-api.git
+
+ # execute the mega-patch (it will post via CURL to localhost:8181)
+ bash ~/path/to/aether-roc-api/examples/MEGA_Patch.curl
+
+
+You may wish to customize the mega patch.
+For example, by default the patch configures the sdcore-adapter to push to sdcore-test-dummy.
+You could configure it to push to a live aether-in-a-box core by doing something like this::
+
+ sed -i 's^http://aether-roc-umbrella-sdcore-test-dummy/v1/config/5g^http://webui.omec.svc.cluster.local:9089/config^g' MEGA_Patch.curl
+
+ #apply the patch
+ ./MEGA_Patch.curl
+
+(Note that if your Aether-in-a-Box was installed on a different machine that port-forwarding may be necessary)
+
+
+Expected CURL output from a successful mega-patch post will be a UUID.
+You can also verify that the mega-patch was successful by going into the aether-roc-gui in a browser
+(see the section on useful port-forwards below). The GUI may open to a dashboard that is unpopulated -- you
+can use the dropdown menu (upper-right hand corner of the screen) to select an object such as VCS and you
+will see a list of VCS.
+
+ |ROCGUI|
+
+Uninstalling the Aether-Roc-Umbrella Helm chart
+-----------------------------------------------
+
+To tear things back down, usually as part of a developer loop prior to redeploying again, do the following::
+
+ helm -n micro-onos del aether-roc-umbrella
+
+If the uninstall hangs or if a subsequent reinstall hangs, it could be an issue with some of the CRDs
+not getting cleaned up. The following may be useful::
+
+ # fix stuck finalizers in operator CRDs
+
+ kubectl -n micro-onos patch entities connectivity-service-v2 --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
+
+ kubectl -n micro-onos patch entities connectivity-service-v3 --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
+
+ kubectl -n micro-onos patch kind aether --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
+
+Useful port forwards
+--------------------
+
+Port forwarding is often necessary to allow access to ports inside of Kubernetes pods that use ClusterIP addressing.
+Note that you typically need to leave a port-forward running (you can put it in the background).
+Also, If you redeploy the ROC and/or if a pod crashes then you might have to restart a port-forward.
+The following port-forwards may be useful::
+
+ # aether-roc-api
+
+ kubectl -n micro-onos port-forward service/aether-roc-api --address 0.0.0.0 8181:8181
+
+ # aether-roc-gui
+
+ kubectl -n micro-onos port-forward service/aether-roc-gui --address 0.0.0.0 8183:80
+
+ # grafana
+
+ kubectl -n micro-onos port-forward service/aether-roc-umbrella-grafana --address 0.0.0.0 8187:80
+
+ # onos gui
+
+ kubectl -n micro-onos port-forward service/onos-gui --address 0.0.0.0 8182:80
+
+Aether-roc-api and aether-roc-gui are in our experience the most useful two port-forwards.
+Aether-roc-api is useful to be able to POST REST API requests.
+Aether-roc-gui is useful to be able to interactively browse the current configuration.
+
+Deploying using custom images
+-----------------------------
+
+Custom images may be used by editing the values-override.yaml file.
+For example, to deploy a custom sdcore-adapter::
+
+ sdcore-adapter-v3:
+
+ prometheusEnabled: false
+
+ image:
+
+ repository: my-private-repo/sdcore-adapter
+
+ tag: my-tag
+
+ pullPolicy: Always
+
+The above example assumes you have published a docker images at my-private-repo/sdcore-adapter:my-tag.
+My particular workflow is to deploy a local-docker registry and push my images to that.
+Please do not publish ONF images to a public repository unless the image is intended to be public.
+Several ONF repositories are private, and therefore their docker artifacts should also be private.
+
+There are alternatives to using a private docker repository.
+For example, if you are using kubadm, then you may be able to simply tag the image locally.
+If you’re using KinD, then you can push a local image to into the kind cluster::
+
+ kind load docker-image sdcore-adapter:my-tag
+
+Inspecting logs
+---------------
+
+Most of the relevant Kubernetes pods are in the micro-onos namespace.
+The names may change from deployment to deployment, so start by getting a list of pods::
+
+ kubectl -n micro-onos get pods
+
+Then you can inspect a specific pod/container::
+
+ kubectl -n micro-onos logs sdcore-adapter-v3-7468cc58dc-ktctz sdcore-adapter-v3
+
+Some exercises to get familiar
+------------------------------
+
+1) Deploy the ROC and POST the mega-patch, go into the aether-roc-GUI and click through the VCS, DeviceGroup, and
+other objects to see that they were created as expected.
+
+2) Examine the log of the sdcore-adapter-v3 container.
+It should be attempting to push the mega-patch’s changes.
+If you don’t have a core available, it may be failing the push, but you should see the attempts.
+
+3) Change an object in the GUI.
+Watch the sdcore-adapter-v3 log file and see that the adapter attempts to push the change.
+
+4) Try POSTing a change via the API.
+Observe the sdcore-adapter-v3 log file and see that the adapter attempts to push the change.
+
+5) Deploy a 5G Aether-in-a-Box (See sd-core developer guide), modify the mega-patch to specify the URL for the
+Aether-in-a-Box webui container, POST the mega-patch, and observe that the changes were correctly pushed via the
+sdcore-adapter-v3 into the sd-core’s webui container (webui container log will show configuration as it is
+received)
+
+.. |ROCGUI| image:: images/rocgui.png
diff --git a/developer/sdcore.rst b/developer/sdcore.rst
new file mode 100644
index 0000000..385f131
--- /dev/null
+++ b/developer/sdcore.rst
@@ -0,0 +1,87 @@
+.. vim: syntax=rst
+
+Aether SD-Core Developer Guide
+==============================
+
+Clone Repositories
+------------------
+
+Clone the following repositories, using your gerrit ids as necessary::
+
+ mkdir -p ~/cord #need to clone below 2 repos in this directory
+ git clone "ssh://<username>@gerrit.opencord.org:29418/helm-charts"
+ git clone "ssh://<username>@gerrit.opencord.org:29418/aether-helm-charts"
+
+ cd ~ #go back to home directory
+ git clone "ssh://<username>@gerrit.opencord.org:29418/aether-in-a-box"
+
+Start the 4G Core
+-----------------
+
+Deploy Aether-in-a-Box and start the 4G core::
+
+ cd aether-in-a-box
+
+ #create K8s cluster for you and also start 4G network functions and carry out 1 ping test
+ make test
+
+Apply the necessary pull secrets
+--------------------------------
+
+The deployment may fail and leave pods in ImagePullBackoff status if the
+necessary secrets are not applied.
+Obtain the secret from ONF, and then execute the following::
+
+ cat > ~/registry-secret.json
+ ... paste the secret here ...
+ kubectl apply -f ~/registry-secret.json
+
+
+
+Transition from 4G Core to 5G Core
+----------------------------------
+
+Before starting the 5G core, the 4G Core must be stopped::
+
+ make reset-test #this does not destroy K8s cluster.
+
+Now, start the 5G Core::
+
+ make 5gc #this will start all 5G network functions with default images.
+
+ kubectl get pods -n omec #verify that all 5g pods are started.
+
+Sample output::
+
+ NAME READY STATUS RESTARTS AGE
+ amf-7d6f649b4f-jzd9r 1/1 Running 0 41s
+ ausf-8dd5d7465-xrjdl 1/1 Running 0 41s
+ gnbsim-0 1/1 Running 0 25s
+ mongodb-55555bc884-wqcrh 1/1 Running 0 41s
+ nrf-ddd7d8d5b-vjv84 1/1 Running 0 41s
+ nssf-7978bb74cc-zct9v 1/1 Running 0 41s
+ pcf-5bbbff96d6-l54f6 1/1 Running 0 41s
+ smf-86b6fd5674-tq8h9 1/1 Running 0 41s
+ udm-5fdf9cf56d-76hlt 1/1 Running 0 41s
+ udr-d8f855c6b-jj8lw 1/1 Running 0 41s
+ ue-setup-if-mftrm 0/1 Completed 0 23d
+ ue-teardown-if-whbvm 0/1 Completed 0 23d
+ upf-0 4/4 Running 0 55s
+ webui-79c8b7dfc7-wn6st 1/1 Running 0 41s
+
+Developer Loop
+--------------
+
+Deploy custom images by editing `~/aether-in-a-box/aether-in-a-box-values.yaml`::
+
+ images:
+ tags:
+ webui: registry.aetherproject.org/omecproject/5gc-webui:onf-release3.0.5-roc-935305f
+ pullPolicy: IfNotPresent
+ pullSecrets:
+ - name: "aether.registry"
+
+Stop the 5G Core and start it again::
+
+ make reset-5g-test
+ make 5gc
diff --git a/edge_deployment/vpn_bootstrap.rst b/edge_deployment/vpn_bootstrap.rst
index 36e3ad9..3d7f7b0 100644
--- a/edge_deployment/vpn_bootstrap.rst
+++ b/edge_deployment/vpn_bootstrap.rst
@@ -106,6 +106,19 @@
If you are adding another ACE to an existing VPN connection, go to
:ref:`Add ACE to an existing VPN connection <add_ace_to_vpn>`
+Get access to encrypted files in aether-pod-configs repository
+--------------------------------------------------------------
+
+`git-crypt <https://github.com/AGWA/git-crypt>`_ is used to securely store encrypted files
+in the aether-pod-configs repository. Before proceeding, (1) install git-crypt and `gpg <https://gnupg.org/>`_,
+(2) create a GPG keypair, and (3) ask a member of the Aether OPs team add your public key
+to the aether-pod-configs keyring. To create the keypair follow these steps:
+
+.. code-block:: shell
+
+ $ gpg --full-generate-key
+ $ gpg --output <key-name>.gpg --armor --export <your-email-address>
+
.. _update_global_resource:
Update global resource maps
@@ -117,6 +130,7 @@
$ cd $WORKDIR
$ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs"
+ $ git-crypt unlock
Add a new ACE information at the end of the following global resource maps.
diff --git a/index.rst b/index.rst
index 73d5499..cf28a3f 100644
--- a/index.rst
+++ b/index.rst
@@ -41,6 +41,7 @@
:glob:
amp/roc
+ amp/monitoring
.. toctree::
:maxdepth: 3
@@ -56,6 +57,15 @@
testing/acceptance_specification
.. toctree::
+ :maxdepth: 3
+ :caption: Aether Developer Notes
+ :hidden:
+ :glob:
+
+ developer/sdcore
+ developer/roc
+
+.. toctree::
:maxdepth: 1
:caption: Meta
:hidden:
