blob: b241d794f27e8e4d5426ee588289c90f9c72af49 [file] [log] [blame]
Scott Bakera91cbd52021-07-28 09:23:08 -07001..
2 SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
3 SPDX-License-Identifier: Apache-2.0
4
5Runtime Operational Control (ROC)
6=================================
7
8Purpose
9-------
10
11The Aether Runtime Operation Control (ROC) is a component designed with the primary purpose of managing the
12Aether Connectivity Service (ACS), including facilitating the integration of edge services with the ACS.
13The Aether ROC allows enterprises to configure subscribers and profiles, as well as implement policies related
14to those profiles. It also allows the Aether operations team to configure the parameters of those policies.
15The ROC is one of many subsystems that make up the Aether Management Platform (AMP).
16
17What the ROC *does* do:
18
19- Add/Update/Delete/Query configuration
20
21- Persist configuration
22
23- Push configuration to services and devices
24
25- Make observations actionable, either manually or automatically
26
27What the ROC *does not* do:
28
29- The ROC does not directly deploy or manage the lifecycle of containers.
30 This is done using the Terraform/Rancher/Helm/Kubernetes stack.
31
32- The ROC does not directly collect or store logging or metric information.
33 This is done using the ElasticStack and Grafana/Prometheus components.
34
35- The ROC is not a message bus used for component-to-component communication.
36 If a message bus is required, then a suitable service such as Kafka could be used.
37
38- The ROC does not implement a service dependency graph.
39 This can be done through helm charts, which are typically hierarchical in nature.
40
41- The ROC is not a formal service mesh.
42 Other tools, such as Istio, could be leveraged to provide service meshes.
43
44- The ROC does not configure *Edge Services*.
Zack Williams1ae109e2021-07-27 11:17:04 -070045 While the ROC's modeling support is general and could be leveraged to support an edge service, and an
Scott Bakera91cbd52021-07-28 09:23:08 -070046 adapter could be written to configure an edge service, promoting an edge service to ROC management would
47 be the exception rather than the rule. Edge services have their own GUIs and APIs, perhaps belonging to
48 a 3rd-party service provider.
49
Zack Williams1ae109e2021-07-27 11:17:04 -070050Although we call out the tasks that ROC doesn't do itself, it's often still necessary for the ROC to be aware
Scott Bakera91cbd52021-07-28 09:23:08 -070051of the actions these other components have taken.
Zack Williams1ae109e2021-07-27 11:17:04 -070052For example, while the ROC doesn't implement a service dependency graph, it is the case that the ROC is aware
Scott Bakera91cbd52021-07-28 09:23:08 -070053of how services are related. This is necessary because some of the actions it takes affect multiple services
54(e.g., a ROC-supported operation on a subscriber profile might result in the ROC making calls to SD-Core,
55SD-RAN, and SD-Fabric).
56
57Throughout the design process, the ROC design team has taken lessons learned from prior systems, such as XOS,
58and applied them to create a next generation design that focuses on solving the configuration problem in a
59focused and lightweight manner.
60
61Design and Requirements
62-----------------------
63
64- The ROC must offer an *API* that may be used by administrators, as well as external services, to configure
65 Aether.
66
67- This ROC API must support new end-to-end abstractions that cross multiple subsystems of Aether.
Zack Williams1ae109e2021-07-27 11:17:04 -070068 For example, "give subscriber X running application Y QoS guarantee Z" is an abstraction that potentially
Scott Bakera91cbd52021-07-28 09:23:08 -070069 spans SD-RAN, SD-Fabric.
70 The ROC defines and implements such end-to-end abstractions.
71
72- The ROC must offer an *Operations GUI* to Operations Personnel, so they may configure the Aether Connectivity
73 service.
74
75- The ROC must offer an *Enterprise GUI* to Enterprise Personnel, so they may configure the connectivity aspects
76 of their particular edge site.
Zack Williams1ae109e2021-07-27 11:17:04 -070077 It's possible this GUI shares implementation with the Operations GUI, but the presentation, content, and
Scott Bakera91cbd52021-07-28 09:23:08 -070078 workflow may differ.
79
80- The ROC must support *versioning* of configuration, so changes can be rolled back as necessary, and an audit
81 history may be retrieved of previous configurations.
82
83- The ROC must support best practices of *performance*, *high availability*, *reliability*, and *security*.
84
85- The ROC must support *role-based access controls (RBAC)*, so that different parties have different visibility
86 into the data model.
87
88- The ROC must be extensible.
89 Aether will incorporate new services over time, and existing services will evolve.
90
91Data Model
92----------
93
94An important aspect of the ROC is that it maintains a data model that represents all the abstractions, such as
95subscribers and profiles, it is responsible for.
Zack Williams1ae109e2021-07-27 11:17:04 -070096The ROC's data model is based on YANG specifications.
Scott Bakera91cbd52021-07-28 09:23:08 -070097YANG is a rich language for data modeling, with support for strong validation of the data stored in the models.
98YANG allows relations between objects to be specified, adding a relational aspect that our previous approaches
99(for example, protobuf) did not directly support.
100YANG is agnostic as to how the data is stored, and is not directly tied to SQL/RDBMS or NoSQL paradigms.
101
102ROC uses tooling built around aether-config (an ONOS-based microservice) to maintain a set of YANG models.
103Among other things, aether-config implements model versioning.
104Migration from one version of the data model to another is supported, as is simultaneous operation of
105different versions.
106
107Architecture
108------------
109
110Below is a high-level architectural diagram of the ROC:
111
112.. image:: images/aether-architecture.svg
Sean Condon257687f2021-08-23 11:13:20 +0100113 :width: 1000
Scott Bakera91cbd52021-07-28 09:23:08 -0700114
115The following walks through the main stack of ROC components in a top-down manner, starting with the GUI(s) and
116ending with the devices/services.
117
118Operations Portal / Enterprise Portal
119"""""""""""""""""""""""""""""""""""""
120
121The code base for the Operations Portal and Enterprise Portal is shared.
122They are two different perspectives of the same portal.
123The *Operations Portal* presents a rougher, more expansive view of the breadth of the Aether modeling.
124The Enterprise Portal presents a more curated view of the modeling.
125These different perspectives can be enforced through the following:
126
127- RBAC controls, to limit access to information that might be unsuitable for a particular party.
128
129- Dashboards, to aggregate/present information in an intuitive manner
130
131- Multi-step workflows (aka Wizards) to break a complex task into smaller guided steps.
132
133The *Portal* is an angular-based typescript GUI.
Zack Williams1ae109e2021-07-27 11:17:04 -0700134The GUI uses REST API to communicate with the ``aether-roc-api`` layer, which in turn communicates with aether-config
Scott Bakera91cbd52021-07-28 09:23:08 -0700135via gNMI.
136The GUI implementation is consistent with modern GUI design, implemented as a single-page application and includes
Zack Williams1ae109e2021-07-27 11:17:04 -0700137a "commit list" that allows several changes to be atomically submitted together.
Scott Bakera91cbd52021-07-28 09:23:08 -0700138Views within the GUI are handcrafted, and as new models are added to Aether, the GUI must be adapted to incorporate
139the new models.
140
141The Portal is a combination of control and observation.
142The control aspect relates to pushing configuration, and the observation aspect relates to viewing metrics,
143logging, and alerts.
144The Portal will leverage other components to do some of the heavy lifting.
145For example, it would make no sense for us to implement our own graph-drawing tool or our own metrics querying
146language when Grafana and Prometheus are already able to do that and we can leverage them.
147GUI pages can be constructed that embed the Grafana renderer.
148
Zack Williams1ae109e2021-07-27 11:17:04 -0700149``aether-roc-api``
150""""""""""""""""""
Scott Bakera91cbd52021-07-28 09:23:08 -0700151
Zack Williams1ae109e2021-07-27 11:17:04 -0700152``aether-roc-api`` a REST API layer that sits between the portals and aether-config.
153The southbound layer of ``aether-roc-api`` is gNMI.
154This is how ``aether-roc-api`` talks to aether-config.
155``aether-roc-api`` at this time is entirely auto-generated; developers need not spend time manually creating REST APIs
Scott Bakera91cbd52021-07-28 09:23:08 -0700156for their models.
157The API layer serves multiple purposes:
158
159- gNMI is an inconvenient interface to use for GUI design, and REST is expected for GUI development.
160
161- The API layer is a potential location for early validation and early security checking, allowing errors to be caught
162 closer to the user.
163 This allows error messages to be generated in a more customary way than gNMI.
164
165- The API layer is yet another place for semantic translation to take place.
166 Although the API layer is currently auto-generated, it is possible that additional methods could be added.
Zack Williams1ae109e2021-07-27 11:17:04 -0700167 gNMI supports only "GET" and "SET", whereas the ``aether-roc-api`` natively supports "GET", "PUT", "POST", "PATCH",
168 and "DELETE".
Scott Bakera91cbd52021-07-28 09:23:08 -0700169
170aether-config
171"""""""""""""
172
Zack Williams1ae109e2021-07-27 11:17:04 -0700173*Aether-config* (a Aether-specific deployment of the "\ *onos-config*\ " microservice) is the core of the ROC's
Scott Bakera91cbd52021-07-28 09:23:08 -0700174configuration system.
175Aether-config is a component that other teams may use in other contexts.
Zack Williams1ae109e2021-07-27 11:17:04 -0700176It's possible that an Aether deployment might have multiple instances of aether-config used for independent purposes.
Scott Bakera91cbd52021-07-28 09:23:08 -0700177The job of aether-config is to store and version configuration data.
178Configuration is pushed to aether-config through the northbound gNMI interface, is stored in an Atomix database
179(not shown in the figure), and is pushed to services and devices using a southbound gNMI interface.
180
181Adapters
182""""""""
183
184Not every device or service beneath the ROC supports gNMI, and in the case where it is not supported, an adapter is
Zack Williams1ae109e2021-07-27 11:17:04 -0700185written to translate between gNMI and the device's or service's native API.
186For example, a gNMI → REST adapter exists to translate between the ROC's modeling and the Aether Connectivity
Scott Bakera91cbd52021-07-28 09:23:08 -0700187Control (SD-Core) components. The adapter is not necessarily only a syntactic translation, but may also be a
188semantic translation.
189[1]_ This supports a logical decoupling of the models stored in the ROC and the interface used by the southbound
190device/service, allowing the southbound device/service and the ROC to evolve independently.
191It also allows for southbound devices/services to be replaced without affecting the northbound interface.
192
193Workflow Engine
194"""""""""""""""
195
196The workflow engine, to the left of the aether-config stack, is where multi-step workflows may be implemented.
197At this time we do not have these workflows, but during the experience with SEBA/VOLTHA, we learned that workflow
198became a key aspect of the implementation.
199For example, SEBA had a state machine surrounding how devices were authorized, activated, and deactivated.
200The workflow engine is a placeholder where workflows may be implemented in Aether as they are required.
201
202Another use of the workflow engine may be to translate between levels in modeling.
203For example, the workflow engine may examine the high-level Enterprise modeling and make changes to the Operations
204modeling to achieve the Enterprise behavior.
205
Zack Williams1ae109e2021-07-27 11:17:04 -0700206Previously this component was referred to as "onos-ztp".
Scott Bakera91cbd52021-07-28 09:23:08 -0700207It is expected that a workflow engine would both read and write the aether-config data model, as well as respond to
208external events.
209
210Analytics Engine
211""""""""""""""""
212
213The analytics engine, to the right of the aether-config stack, is where enrichment of analytics will be performed.
214Raw metrics and logs are collected with open source components Grafana/Prometheus and ElasticStack.
215Those metrics might need additional transformation before they can be presented to Enterprise users, or in some
216cases even before they are presented to the Ops team.
217The Analytics engine would be a place where those metrics could be transformed or enriched, and then written back
218to Prometheus or Elastic (or forwarded as alerts).
219
220The analytics engine is also where analytics would be related to config models in aether-config, in order for
221Enterprise or Operations personnel to take action in response to data and insights received through analytics.
Zack Williams1ae109e2021-07-27 11:17:04 -0700222Action doesn't necessarily have to involve humans.
Scott Bakera91cbd52021-07-28 09:23:08 -0700223It is expected that the combination of Analytics Engine and Workflow Engine could automate a response.
224
225The analytics engine also provides an opportunity to implement access control from the telemetry API.
226Prometheus itself is not multi-tenant and does not support fine-grained access controls.
227
228Aether Operator
229"""""""""""""""
230
231Not pictured in the diagram is the ONOS Operator, which is responsible for configuring the models within
232aether-config. Models to load are specified by a helm chart.
233The operator compiles them on demand and incorporates them into aether-config.
234This eliminates dynamic load compatibility issues that were previously a problem with building models and
235aether-config separately. Operators are considered a best practice in Kubernetes.
236
237Modules are loaded into the process primarily for performance and simplicity reasons.
Zack Williams1ae109e2021-07-27 11:17:04 -0700238The design team has had experience with other systems (for example, VOLTHA and XOS) where modules were decoupled
Scott Bakera91cbd52021-07-28 09:23:08 -0700239and message buses introduced between them, but that can lead to both complexity issues and performance bottlenecks
Zack Williams1ae109e2021-07-27 11:17:04 -0700240in those systems. The same module and operator pattern will be applied to ``aether-roc-api``.
Scott Bakera91cbd52021-07-28 09:23:08 -0700241
242Aether Modeling
243---------------
244
245There is no fixed distinction between high-level and low-level modeling in the ROC.
246There is one set of Aether modeling that might have customer-facing and internal-facing aspects.
247
248.. image:: images/aether-highlevel.svg
Sean Condon257687f2021-08-23 11:13:20 +0100249 :width: 600
Scott Bakera91cbd52021-07-28 09:23:08 -0700250
251The above diagram is an example of how a single set of models could serve both high-level and low-level needs and
252is not necessarily identical to the current implementation.
253For example, *App* and *Service* are concepts that are necessarily enterprise-facing.
254*UPF*\ s are concepts that are operator-facing.
255A UPF might be used by a Service, but the customer need not be aware of this detail.
256Similarly, some objects might be partially customer-facing and partially operator-facing.
257For example, a *Radio* is a piece of hardware the customer has deployed on his premises, so he must know of it, but
258the configuration details of the radio (signal strength, IP address, etc) are operator-facing.
259
260An approximation of the current Aether-3.0 (Release 1.5) modeling is presented below:
261
262.. image:: images/aether-3.0-models.svg
Sean Condon257687f2021-08-23 11:13:20 +0100263 :width: 800
Scott Bakera91cbd52021-07-28 09:23:08 -0700264
265The key Enterprise-facing abstractions are *Applicatio*\ n, *Virtual Cellular Service* (VCS), and *DeviceGroup*.
266
267Identity Management
268-------------------
269
270The ROC leverages an external identity database (i.e.
271LDAP server) to store user data such as account names and passwords for users who are able to log in to the ROC.
272This LDAP server also has the capability to associate users with groups, for example adding ROC administrators to
273ONFAetherAdmin would be a way to grant those people administrative privileges within the ROC.
274
275An external authentication service (DEX) is used to authenticate the user, handling the mechanics of accepting the
276password, validating it, and securely returning the group the user belongs to.
277The group identifier is then used to grant access to resources within the ROC.
278
279The ROC leverages Open Policy Agent (OPA) as a framework for writing access control policies.
280
281Securing Machine-to-Machine Communications
282------------------------------------------
283
284gNMI naturally lends itself to mutual TLS for authentication, and that is the recommended way to secure
285communications between components that speak gNMI.
286For example, the communication between aether-config and its adapters uses gNMI and therefore uses mutual TLS.
287Distributing certificates between components is a problem outside the scope of the ROC.
Zack Williams1ae109e2021-07-27 11:17:04 -0700288It's assumed that another tool will be responsible for distribution, renewing certificates before they expire, etc.
Scott Bakera91cbd52021-07-28 09:23:08 -0700289
290For components that speak REST, HTTPS is used to secure the connection, and authentication can take place using
291mechanisms within the HTTPS protocol (basic auth, tokens, etc).
292Oath2 and OpenID Connect are leveraged as an authorization provider when using these REST APIs.
293
294.. [1]
Zack Williams1ae109e2021-07-27 11:17:04 -0700295 Adapters are an ad hoc approach to implementing the workflow engine,
Scott Bakera91cbd52021-07-28 09:23:08 -0700296 where they map models onto models, including the appropriate semantic
297 translation. This is what we originally did in XOS, but we prefer a
298 more structured approach for ROC.
299
300
Sean Condon256df682022-01-24 14:36:16 +0000301Operations Portal Usage
302-----------------------
Scott Bakera91cbd52021-07-28 09:23:08 -0700303
Sean Condon256df682022-01-24 14:36:16 +0000304The Operations Portal is available as a web application, at a location defined in the Ingress of the Cluster.
305
306It is secured by SSL and an authentication system based on OpenID Connect. The implementation of this is through
307Keycloak, with users and groups defined in LDAP. It has a Role Based Access Control (RBAC) implementation based
308on Open Policy Agent (OPA).
309
310The Operations Portal is built on Angular 12 framework, and is compatible with the latest versions of:
311
312.. list-table:: Browser Compatibility
313 :widths: 40 60
314 :header-rows: 0
315
316 * - Google Chrome
317 - latest
318 * - Mozilla Firefox
319 - latest and extended support release (ESR)
320 * - Microsoft Edge
321 - 2 most recent major versions
322 * - Apple Safari
323 - 2 most recent major versions
324 * - Apple iOS
325 - 2 most recent major versions
326 * - Google Android
327 - 2 most recent major versions
328
329Logging in to the portal
330""""""""""""""""""""""""
331Navigating to the portal URL e.g. ``https://roc.<cluster-ingress-hostname>`` will briefly show
332an empty dashboard, before temporarily redirecting to the Authentication server's login
333page e.g. ``https://keycloak.ananki.io``
334
335.. image:: images/keycloak-login.png
336 :width: 800
337
338.. note:: The username and password are your Keycloak credentials - if you have forgotten
339 your password, please contact the system administrator.
340
341On entering the "username" and "password" you will be redirected to the dashboard.
342
343Clicking the user name in the drop down menu will display your name and the groups you belong to:
344
345.. image:: images/ops-portal-login-details.png
346 :width: 800
347
348.. note:: Click on the panel to dismiss it
349
350Clicking on the API Key, shows your Session API Key, along with the time it
351expires (in 24 hours time).
352
353.. image:: images/ops-portal-api-key.png
354 :width: 800
355
356Logging out of the portal
357"""""""""""""""""""""""""
358
359To log out of the portal click the "Sign Out" button at the end of the drop down menu.
360This releases all resources and redirects you to the Keycloak Account Management page,
361where you will have to click **Sign Out** to terminate the Keycloak session.
362
363.. image:: images/keycloak-account-management.png
364 :width: 800
365
366Before you "Sign Out", while still in this screen, it is possible to:
367
368* inspect your **Personal Info**,
369* configure ways to sign in
370* set up 2 factor authentication
371* change your password if required.
372* see your device activity
373
374.. note:: To return to the Operations Portal you can simply browse to the Portal's URL, or use the
375 browser's back button to return to the dashboard.