Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 1 | .. vim: syntax=rst |
| 2 | |
| 3 | Aether ROC Developer Guide |
| 4 | ========================== |
| 5 | |
| 6 | Background / Development Environment |
| 7 | ------------------------------------ |
| 8 | |
| 9 | This document assumes familiarity with Kubernetes and Helm, and that a Kubernetes/Helm development |
| 10 | environment has already been deployed in the developer’s work environment. |
| 11 | This development environment can use any of a number of potential mechanisms -- including KinD, Kubeadm, etc. |
| 12 | The Aether-in-a-Box script is one potential way to setup a development environment, but not the only way. |
| 13 | As an alternative to the developer’s local machine, a remote environment can be set up, for example on |
| 14 | cloud infrastructure such as cloudlab. |
| 15 | |
Sean Condon | eb95cd6 | 2021-08-04 19:44:18 +0100 | [diff] [blame] | 16 | .. note:: When ROC is deployed it is unsecured by default, with no Authentication or Authorization. |
| 17 | To secure ROC so that the Authentication and Authorization can be tested, follow the Securing ROC |
| 18 | guide below :ref:`securing_roc` |
| 19 | |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 20 | Installing Prerequisites |
| 21 | ------------------------ |
| 22 | |
| 23 | Atomix and onos-operator must be installed:: |
| 24 | |
| 25 | # create necessary namespaces |
| 26 | kubectl create namespace micro-onos |
| 27 | |
| 28 | # install atomix |
| 29 | helm -n kube-system install atomix-controller atomix/atomix-controller |
| 30 | helm -n kube-system install atomix-raft-storage atomix/atomix-raft-storage |
| 31 | |
| 32 | # install the onos operator |
| 33 | helm install -n kube-system onos-operator onosproject/onos-operator |
| 34 | |
| 35 | |
| 36 | Verify that these services were installed properly. |
| 37 | You should see pods for *atomix-controller*, *atomix-raft-storage-controller*, |
| 38 | *onos-operator-config*, and *onos-operator-topo*. |
| 39 | Execute these commands:: |
| 40 | |
| 41 | kubectl -n kube-system get pods | grep -i atomix |
| 42 | kubectl -n kube-system get pods | grep -i onos |
| 43 | |
| 44 | |
| 45 | Create a values-override.yaml |
| 46 | ----------------------------- |
| 47 | |
| 48 | You’ll want to override several of the defaults in the ROC helm charts:: |
| 49 | |
| 50 | cat > values-override.yaml <<EOF |
| 51 | import: |
Scott Baker | b46a6ed | 2021-08-02 14:03:10 -0700 | [diff] [blame] | 52 | onos-gui: |
| 53 | enabled: true |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 54 | |
| 55 | onos-gui: |
Scott Baker | b46a6ed | 2021-08-02 14:03:10 -0700 | [diff] [blame] | 56 | ingress: |
| 57 | enabled: false |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 58 | |
| 59 | aether-roc-gui-v3: |
Scott Baker | b46a6ed | 2021-08-02 14:03:10 -0700 | [diff] [blame] | 60 | ingress: |
| 61 | enabled: false |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 62 | EOF |
| 63 | |
| 64 | Installing the Aether-Roc-Umbrella Helm chart |
| 65 | --------------------------------------------- |
| 66 | |
| 67 | Add the necessary helm repositories:: |
| 68 | |
| 69 | # obtain username and password from Michelle and/or ONF infra team |
| 70 | export repo_user=<username> |
| 71 | export repo_password=<password> |
| 72 | helm repo add sdran --username "$repo_user" --password "$repo_password" https://sdrancharts.onosproject.org |
| 73 | |
| 74 | Aether-Roc-Umbrella will bring up the ROC and its services:: |
| 75 | |
| 76 | helm -n micro-onos install aether-roc-umbrella sdran/aether-roc-umbrella -f values-override.yaml |
| 77 | |
| 78 | kubectl wait pod -n micro-onos --for=condition=Ready -l type=config --timeout=300s |
| 79 | |
| 80 | |
Sean Condon | f918f64 | 2021-08-04 14:32:53 +0100 | [diff] [blame] | 81 | .. _posting-the-mega-patch: |
| 82 | |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 83 | Posting the mega-patch |
| 84 | ---------------------- |
| 85 | |
| 86 | The ROC usually comes up in a blank state -- there are no Enterprises, UEs, or other artifacts present in it. |
| 87 | The mega-patch is an example patch that populates the ROC with some sample enterprises, UEs, slices, etc. |
| 88 | Execute the following:: |
| 89 | |
| 90 | # launch a port-forward for the API |
| 91 | # this will continue to run in the background |
| 92 | kubectl -n micro-onos port-forward service/aether-roc-api --address 0.0.0.0 8181:8181 & |
| 93 | |
| 94 | git clone https://github.com/onosproject/aether-roc-api.git |
| 95 | |
| 96 | # execute the mega-patch (it will post via CURL to localhost:8181) |
| 97 | bash ~/path/to/aether-roc-api/examples/MEGA_Patch.curl |
| 98 | |
| 99 | |
| 100 | You may wish to customize the mega patch. |
| 101 | For example, by default the patch configures the sdcore-adapter to push to sdcore-test-dummy. |
| 102 | You could configure it to push to a live aether-in-a-box core by doing something like this:: |
| 103 | |
| 104 | sed -i 's^http://aether-roc-umbrella-sdcore-test-dummy/v1/config/5g^http://webui.omec.svc.cluster.local:9089/config^g' MEGA_Patch.curl |
| 105 | |
| 106 | #apply the patch |
| 107 | ./MEGA_Patch.curl |
| 108 | |
| 109 | (Note that if your Aether-in-a-Box was installed on a different machine that port-forwarding may be necessary) |
| 110 | |
| 111 | |
| 112 | Expected CURL output from a successful mega-patch post will be a UUID. |
| 113 | You can also verify that the mega-patch was successful by going into the aether-roc-gui in a browser |
| 114 | (see the section on useful port-forwards below). The GUI may open to a dashboard that is unpopulated -- you |
| 115 | can use the dropdown menu (upper-right hand corner of the screen) to select an object such as VCS and you |
| 116 | will see a list of VCS. |
| 117 | |
| 118 | |ROCGUI| |
| 119 | |
| 120 | Uninstalling the Aether-Roc-Umbrella Helm chart |
| 121 | ----------------------------------------------- |
| 122 | |
| 123 | To tear things back down, usually as part of a developer loop prior to redeploying again, do the following:: |
| 124 | |
| 125 | helm -n micro-onos del aether-roc-umbrella |
| 126 | |
| 127 | If the uninstall hangs or if a subsequent reinstall hangs, it could be an issue with some of the CRDs |
| 128 | not getting cleaned up. The following may be useful:: |
| 129 | |
| 130 | # fix stuck finalizers in operator CRDs |
| 131 | |
| 132 | kubectl -n micro-onos patch entities connectivity-service-v2 --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' |
| 133 | |
| 134 | kubectl -n micro-onos patch entities connectivity-service-v3 --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' |
| 135 | |
| 136 | kubectl -n micro-onos patch kind aether --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' |
| 137 | |
| 138 | Useful port forwards |
| 139 | -------------------- |
| 140 | |
| 141 | Port forwarding is often necessary to allow access to ports inside of Kubernetes pods that use ClusterIP addressing. |
| 142 | Note that you typically need to leave a port-forward running (you can put it in the background). |
| 143 | Also, If you redeploy the ROC and/or if a pod crashes then you might have to restart a port-forward. |
| 144 | The following port-forwards may be useful:: |
| 145 | |
| 146 | # aether-roc-api |
| 147 | |
| 148 | kubectl -n micro-onos port-forward service/aether-roc-api --address 0.0.0.0 8181:8181 |
| 149 | |
| 150 | # aether-roc-gui |
| 151 | |
| 152 | kubectl -n micro-onos port-forward service/aether-roc-gui --address 0.0.0.0 8183:80 |
| 153 | |
| 154 | # grafana |
| 155 | |
| 156 | kubectl -n micro-onos port-forward service/aether-roc-umbrella-grafana --address 0.0.0.0 8187:80 |
| 157 | |
| 158 | # onos gui |
| 159 | |
| 160 | kubectl -n micro-onos port-forward service/onos-gui --address 0.0.0.0 8182:80 |
| 161 | |
| 162 | Aether-roc-api and aether-roc-gui are in our experience the most useful two port-forwards. |
| 163 | Aether-roc-api is useful to be able to POST REST API requests. |
| 164 | Aether-roc-gui is useful to be able to interactively browse the current configuration. |
| 165 | |
| 166 | Deploying using custom images |
| 167 | ----------------------------- |
| 168 | |
| 169 | Custom images may be used by editing the values-override.yaml file. |
| 170 | For example, to deploy a custom sdcore-adapter:: |
| 171 | |
| 172 | sdcore-adapter-v3: |
| 173 | |
| 174 | prometheusEnabled: false |
| 175 | |
| 176 | image: |
| 177 | |
| 178 | repository: my-private-repo/sdcore-adapter |
| 179 | |
| 180 | tag: my-tag |
| 181 | |
| 182 | pullPolicy: Always |
| 183 | |
| 184 | The above example assumes you have published a docker images at my-private-repo/sdcore-adapter:my-tag. |
| 185 | My particular workflow is to deploy a local-docker registry and push my images to that. |
| 186 | Please do not publish ONF images to a public repository unless the image is intended to be public. |
| 187 | Several ONF repositories are private, and therefore their docker artifacts should also be private. |
| 188 | |
| 189 | There are alternatives to using a private docker repository. |
| 190 | For example, if you are using kubadm, then you may be able to simply tag the image locally. |
| 191 | If you’re using KinD, then you can push a local image to into the kind cluster:: |
| 192 | |
| 193 | kind load docker-image sdcore-adapter:my-tag |
| 194 | |
| 195 | Inspecting logs |
| 196 | --------------- |
| 197 | |
| 198 | Most of the relevant Kubernetes pods are in the micro-onos namespace. |
| 199 | The names may change from deployment to deployment, so start by getting a list of pods:: |
| 200 | |
| 201 | kubectl -n micro-onos get pods |
| 202 | |
| 203 | Then you can inspect a specific pod/container:: |
| 204 | |
| 205 | kubectl -n micro-onos logs sdcore-adapter-v3-7468cc58dc-ktctz sdcore-adapter-v3 |
| 206 | |
Sean Condon | eb95cd6 | 2021-08-04 19:44:18 +0100 | [diff] [blame] | 207 | .. _securing_roc: |
| 208 | |
| 209 | Securing ROC |
| 210 | ------------ |
| 211 | |
| 212 | When deploying ROC with the **aether-roc-umbrella** chart, secure mode can be enabled by |
| 213 | specifying an OpenID Connect (OIDC) issuer like:: |
| 214 | |
| 215 | helm -n micro-onos install aether-roc-umbrella sdran/aether-roc-umbrella \ |
| 216 | --set onos-config.openidc.issuer=http://dex-ldap-umbrella:5556 \ |
| 217 | --set aether-roc-gui-v3.openidc.issuer=http://dex-ldap-umbrella:5556 |
| 218 | |
| 219 | The choice of OIDC issuer in this case is **dex-ldap-umbrella** |
| 220 | |
| 221 | dex-ldap-umbrella |
| 222 | ~~~~~~~~~~~~~~~~~ |
| 223 | |
| 224 | Dex is a cloud native OIDC Issuer than can act as a front end to several authentication systems |
| 225 | e.g. LDAP, Crowd, Google, GitHub |
| 226 | |
| 227 | Dex-LDAP-Umbrella is a Helm chart that combines a Dex server with an LDAP installation, and an |
| 228 | LDAP administration tool. It can be deployed in to the same cluster namespace as **aether-roc-umbrella**. |
| 229 | |
| 230 | Its LDAP server is populated with 7 different users in the 2 example enterprises - *starbucks* and *acme*. |
| 231 | |
| 232 | When running it should be available at *http://dex-ldap-umbrella:5556/.well-known/openid-configuration*. |
| 233 | |
| 234 | See `dex-ldap-umbrella <https://github.com/onosproject/onos-helm-charts/tree/master/dex-ldap-umbrella#readme>`_ |
| 235 | for more details. |
| 236 | |
| 237 | As an alternative there is a public Dex server connected to the ONF Crowd server, that allows |
| 238 | ONF staff to login with their own credentials: |
| 239 | See `public dex <https://dex.aetherproject.org/dex/.well-known/openid-configuration>`_ for more details. |
| 240 | |
| 241 | .. note:: Your RBAC access to ROC will be limited by the groups you belong to in Crowd. |
| 242 | |
| 243 | Role Based Access Control |
| 244 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 245 | When secured, access to the configuration in ROC is limited by the **groups** that a user belongs to. |
| 246 | |
| 247 | * **AetherROCAdmin** - users in this group have full read **and** write access to all configuration. |
| 248 | * *<enterprise>* - users in a group the lowercase name of an enterprise, will have **read** access to that enterprise. |
| 249 | * **EnterpriseAdmin** - users in this group will have read **and** write access the enterprise they belong to. |
| 250 | |
| 251 | For example in *dex-ldap-umbrella* the user *Daisy Duke* belongs to *starbucks* **and** |
| 252 | *EnterpriseAdmin* and so has read **and** write access to items linked with *starbucks* enterprise. |
| 253 | |
| 254 | By comparison the user *Elmer Fudd* belongs only to *starbucks* group and so has only **read** access to items |
| 255 | linked with the *starbucks* enterprise. |
| 256 | |
| 257 | Requests to a Secure System |
| 258 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 259 | When configuration is retrieved or updated through *aether-config*, a Bearer Token in the |
| 260 | form of a Json Web Token (JWT) issued by the selected OIDC Issuer server must accompany |
| 261 | the request as an Authorization Header. |
| 262 | |
| 263 | This applies to both the REST interface of *aether-roc-api* **and** the *gnmi* interface of |
| 264 | *aether-rconfig*. |
| 265 | |
| 266 | In the Aether ROC, a Bearer Token can be generated by logging in and selecting API Key from the |
| 267 | menu. This pops up a window with a copy button, where the key can be copied. |
| 268 | |
| 269 | The key will expire after 24 hours. |
| 270 | |
| 271 | .. image:: images/aether-roc-gui-copy-api-key.png |
| 272 | :width: 580 |
| 273 | :alt: Aether ROC GUI allows copying of API Key to clipboard |
| 274 | |
| 275 | Accessing the REST interface from a tool like Postman, should include this Auth token. |
| 276 | |
| 277 | .. image:: images/postman-auth-token.png |
| 278 | :width: 930 |
| 279 | :alt: Postman showing Authentication Token pasted in |
| 280 | |
| 281 | Logging |
| 282 | ~~~~~~~ |
| 283 | The logs of *aether-config* will contain the **username** and **timestamp** of |
| 284 | any **gnmi** call when security is enabled. |
| 285 | |
| 286 | .. image:: images/aether-config-log.png |
| 287 | :width: 887 |
| 288 | :alt: aether-config log message showing username and timestamp |
| 289 | |
Sean Condon | 435be9a | 2021-08-06 14:28:37 +0100 | [diff] [blame] | 290 | Accessing GUI from an external system |
| 291 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 292 | To access the ROC GUI from a computer outside the Cluster machine using *port-forwarding* then |
| 293 | it is necessary to: |
| 294 | |
| 295 | * Ensure that all *port-forward*'s have **--address=0.0.0.0** |
| 296 | * Add to the IP address of the cluster machine to the **/etc/hosts** of the outside computer as:: |
| 297 | |
| 298 | <ip address of cluster> dex-ldap-umbrella aether-roc-gui |
| 299 | * Verify that you can access the Dex server by its name *http://dex-ldap-umbrella:5556/.well-known/openid-configuration* |
| 300 | * Access the GUI through the hostname (rather than ip address) *http://aether-roc-gui:8183* |
| 301 | |
Sean Condon | eb95cd6 | 2021-08-04 19:44:18 +0100 | [diff] [blame] | 302 | Troubleshooting Secure Access |
| 303 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 304 | While every effort has been made to ensure that securing Aether is simple and effective, |
| 305 | some difficulties may arise. |
| 306 | |
| 307 | One of the most important steps is to validate that the OIDC Issuer (Dex server) can be reached |
| 308 | from the browser. The **well_known** URL should be available and show the important endpoints are correct. |
| 309 | |
| 310 | .. image:: images/dex-ldap-umbrella-well-known.png |
| 311 | :width: 580 |
| 312 | :alt: Dex Well Known page |
| 313 | |
| 314 | If logged out of the Browser when accessing the Aether ROC GUI, accessing any page of the application should |
| 315 | redirect to the Dex login page. |
| 316 | |
| 317 | .. image:: images/dex-ldap-login-page.png |
| 318 | :width: 493 |
| 319 | :alt: Dex Login page |
| 320 | |
| 321 | When logged in the User details can be seen by clicking the User's name in the drop down menu. |
| 322 | This shows the **groups** that the user belongs to, and can be used to debug RBAC issues. |
| 323 | |
| 324 | .. image:: images/aether-roc-gui-user-details.png |
| 325 | :width: 700 |
| 326 | :alt: User Details page |
| 327 | |
| 328 | When you sign out of the ROC GUI, if you are not redirected to the Dex Login Page, |
| 329 | you should check the Developer Console of the browser. The console should show the correct |
| 330 | OIDC issuer (dex server), and that Auth is enabled. |
| 331 | |
| 332 | .. image:: images/aether-roc-gui-console-loggedin.png |
| 333 | :width: 418 |
| 334 | :alt: Browser Console showing correct configuration |
| 335 | |
Scott Baker | b46a6ed | 2021-08-02 14:03:10 -0700 | [diff] [blame] | 336 | ROC Data Model Conventions and Requirements |
| 337 | ------------------------------------------- |
| 338 | |
| 339 | The MEGA-Patch described above will bring up a fully compliant sample data model. |
| 340 | However, it may be useful to bring up your own data model, customized to a different |
| 341 | site of sites. This subsection documents conventions and requirements for the Aether |
| 342 | modeling within the roc. |
| 343 | |
| 344 | The ROC models must be configured with the following: |
| 345 | |
| 346 | * A default enterprise with the id `defaultent`. |
| 347 | * A default ip-domain with the id `defaultent-defaultip`. |
| 348 | * A default site with the id `defaultent-defaultsite`. |
| 349 | This site should be linked to the `defaultent` enterprise. |
| 350 | * A default device group with the id `defaultent-defaultsite-default`. |
| 351 | This device group should be linked to the `defaultent-defaultip` ip-domain |
| 352 | and the `defaultent-defaultsite` site. |
| 353 | |
| 354 | Each Enterprise Site must be configured with a default device group and that default |
| 355 | device group's name must end in the suffix `-default`. For example, `acme-chicago-default`. |
| 356 | |
Scott Baker | fab7c9e | 2021-07-29 17:12:16 -0700 | [diff] [blame] | 357 | Some exercises to get familiar |
| 358 | ------------------------------ |
| 359 | |
| 360 | 1) Deploy the ROC and POST the mega-patch, go into the aether-roc-GUI and click through the VCS, DeviceGroup, and |
| 361 | other objects to see that they were created as expected. |
| 362 | |
| 363 | 2) Examine the log of the sdcore-adapter-v3 container. |
| 364 | It should be attempting to push the mega-patch’s changes. |
| 365 | If you don’t have a core available, it may be failing the push, but you should see the attempts. |
| 366 | |
| 367 | 3) Change an object in the GUI. |
| 368 | Watch the sdcore-adapter-v3 log file and see that the adapter attempts to push the change. |
| 369 | |
| 370 | 4) Try POSTing a change via the API. |
| 371 | Observe the sdcore-adapter-v3 log file and see that the adapter attempts to push the change. |
| 372 | |
| 373 | 5) Deploy a 5G Aether-in-a-Box (See sd-core developer guide), modify the mega-patch to specify the URL for the |
| 374 | Aether-in-a-Box webui container, POST the mega-patch, and observe that the changes were correctly pushed via the |
| 375 | sdcore-adapter-v3 into the sd-core’s webui container (webui container log will show configuration as it is |
| 376 | received) |
| 377 | |
| 378 | .. |ROCGUI| image:: images/rocgui.png |
Sean Condon | eb95cd6 | 2021-08-04 19:44:18 +0100 | [diff] [blame] | 379 | :width: 945 |
| 380 | :alt: ROC GUI showing list of VCS |