blob: 91cf309e5e8c73616e45982df84fb67a8f1010fc [file] [log] [blame]
divyadesaia702ba92020-08-31 11:27:46 +00001Centralized Log Analysis
2========================
3
4Objective
5---------
6
7Operators should be able to view logs from all the VOLTHA components as
8well as from whitebox OLT devices in a single stream.
9
Andrea Campanella882cfcc2021-02-04 10:53:57 +010010Solution Approach For VOLTHA Ecosystem
divyadesaia702ba92020-08-31 11:27:46 +000011--------------------------------------
12
13The solution we have chosen EFK (elasticsearch, kibana and
14fluentd-elasticsearch) setup for voltha enables the Operator to push
15logs from all VOLTHA components.
16
Andrea Campanellac18d1182021-09-10 12:01:38 +020017To deploy VOLTHA with the EFK stack follow the paraghraph `Support-for-logging-and-tracing-(optional)`
18in the `voltha-helm-charts README <../voltha-helm-charts/README.md>`_.
Andrea Campanella882cfcc2021-02-04 10:53:57 +010019
20This will deploy Efk stack with a single node elasticsearch and
divyadesaia702ba92020-08-31 11:27:46 +000021kibana instance will be deployed and a fluentd-elasticsearch pod will be
Andrea Campanella882cfcc2021-02-04 10:53:57 +010022deployed on each node that allows workloads to be scheduled.
divyadesaia702ba92020-08-31 11:27:46 +000023
24The number of deployed Pods will be dependent on the value of Deployment
25Type and SCHEDULE\_ON\_CONTROL\_NODES flag as shown in the below table.
26
27.. figure:: ../_static/fluentd-pods.png
28 :width: 6.50000in
29 :height: 1.50000in
30
divyadesaia702ba92020-08-31 11:27:46 +000031**To start using Kibana, In your browser ,navigate to
32http://<k8s\_node\_ip>:<exposed\_port>.** Then you can search for events
33in the *Discover* section.
34
35Solution Approach For Whitebox OLT Device
36-----------------------------------------
37
38The solution approach we have chosen is to install td-agent (fluentd
39variant) directly on OLT device for capturing and transmitting logs to
40elasticsearch pod running in voltha cluster.
41
42Created custom td-agent configuration file to handle the format of
43involved log files using right input plugins for openolt process, dev
44mgmt daemon, system logs and elasticsearch output plugin.You can find
45custom td-agent configuration file in
46`*https://github.com/opencord/openolt/tree/master/logConf* <https://github.com/opencord/openolt/tree/master/logConf>`__
47and find installation steps in
48`*https://github.com/opencord/openolt/tree/master* <https://github.com/opencord/openolt/tree/master/logConf>`__
49README.
50
51Log Collection from VOLTHA Ecosystem and Whitebox OLT Device
52------------------------------------------------------------
53
54Below diagram depicts the log collection from voltha components and
Andrea Campanella8245ff52021-10-06 11:50:03 +020055whitebox OLT device through EFK.The fluentd pod running collects logs
56from all the voltha components and push to elasticsearch pod.
57The td-agent(fluentd variant) service running on
divyadesaia702ba92020-08-31 11:27:46 +000058whitebox OLT device capture the logs from openolt agent process, device
59mgmt daemon process and system logs and transmits the logs to the
60elasticsearch pod running in voltha cluster over tcp protocol.
61
62.. figure:: ../_static/centralize-logging.png
63 :width: 6.50000in
64 :height: 2.50000in
65
66Secure EFK setup and transport of Logs from OLT device
67------------------------------------------------------
68
69The Operator can enhance the setup by making configuration changes with
70the requirement.
71
72The Authentication, Authorization, and Security features for EFK can be
73enabled via X-Pack plugin and Role Based Access Control (RBAC) in
74Elasticsearch.The transmission of logs from the Whitebox OLT device can
75be secured by enabling tls/ssl encryption with EFK setup and
76td-agent.Refer following link for Security features.
77`*https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-security.html* <https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-security.html>`__
78
79To enable TLS/SSL encryption for elasticsearch pod refer the following
80link
81
82`*https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/security* <https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/security>`__
83
84To enable TLS/SSL encryption for kibana pod refer the following link
85
86`*https://github.com/elastic/helm-charts/tree/master/kibana/examples/security* <https://github.com/elastic/helm-charts/tree/master/kibana/examples/security>`__
87
88To enable TLS/SSL encryption for fluentd pod and td-agent service refer
89following link
90
91`*https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch* <https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch>`__
92
93Note: create certs directory in /etc/td-agent on the OLT device and copy
94the elastic-ca.pem certificate.
95
96Archive of Logs
97---------------
98
99There are various mechanisms available with EFK to save data.For example
100operators can use **reporting feature** to generate reports of saved
101search as CSV documents, that can be transferred to a support
102organization via email.You can save searches with time-boxed or with
103filtering the required fields then generate the report.To use reporting
104features refer the following link
105`*https://www.elastic.co/guide/en/kibana/current/reporting-getting-started.html* <https://www.elastic.co/guide/en/kibana/current/reporting-getting-started.html>`__
106
107Note: By default a 10mb of CSV file can be generated.To generate > 10mb
108of file enable x-pack plugin and rbac.To generate larger files need to
109have a bigger cluster configuration for the elasticsearch pod.The java
110heap space,cpu and memory need to be increased with the CSV file size.