[VOL-3386] Add support for secure gRPC in openolt-agent

The init script of the openolt service may start with '--enable-tls <TLS_OPTION>' argument for the gRPC server.
Default is insecure with no '--enable-tls' argument.
The TLS capability depends upon the certificates stored at the keystore/ directory: 1. root.crt (CA public key), 2. server.crt (public key), 3.server.key (private key).
Four unit tests are added for the secure gRPC server which work with the keystore-test/ directory.
The certificates stored at the keystore-test/ directory are self-signed certificates, valid until Apr 11 23:16:58 2031 GMT.

Change-Id: I4d18a98a0193f501f922360c79f54b0fcedf14a5
diff --git a/.gitignore b/.gitignore
index 2ebaf57..f510ac4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,3 +36,6 @@
 
 # vscode
 .vscode
+
+# for unit tests
+keystore/
diff --git a/README.md b/README.md
index b6c0440..4c98aec 100644
--- a/README.md
+++ b/README.md
@@ -219,6 +219,7 @@
 ### Additional notes
 
 * *9191* is the TCP port that the *OpenOLT* agent uses for its gRPC channel.
+* See the [SECURITY.md](./SECURITY.md) for possible TLS configurations.
 * In the commands above, you should use OLT inband interface IP address if OLT
   is used in inband mode, otherwise substitute all its occurrences with the
   management IP of your OLT.
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 0000000..a7827d3
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,136 @@
+# Security for the GRPC Server with TLS
+
+[gRPC](https://grpc.io/) has built-in support for [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) both for the server and client applications.
+
+From the [documentation](https://www.grpc.io/docs/guides/auth/):
+> gRPC has SSL/TLS integration and promotes the use of SSL/TLS to authenticate the server, and to encrypt all the data exchanged between the client and the server. Optional mechanisms are available for clients to provide certificates for mutual authentication.
+
+## Options for TLS
+
+For the server case, the enumeration [grpc_ssl_client_certificate_request_type](https://grpc.github.io/grpc/cpp/grpc__security__constants_8h.html) can be used how to respond to an incoming gRPC client.
+
+The values of this enumeration are described by the following table:
+
+| Enumerator                               | Description                                  |
+|------------------------------------------|----------------------------------------------|
+| GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE | Server does not request client certificate.<br>The certificate presented by the client is not checked by the server at all.<br>(A client may present a self signed or signed certificate or not present a certificate at all and any of those option would be accepted) |
+| GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY | Server requests client certificate but does not enforce that the client presents a certificate.<br>If the client presents a certificate, the client authentication is left to the application (the necessary metadata will be available to the application via authentication context properties, see grpc_auth_context).<br>The client's key certificate pair must be valid for the SSL connection to be established. |
+| GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY | Server requests client certificate but does not enforce that the client presents a certificate.<br>If the client presents a certificate, the client authentication is done by the gRPC framework.<br>(For a successful connection the client needs to either present a certificate that can be verified against the root certificate configured by the server or not present a certificate at all)<br>The client's key certificate pair must be valid for the SSL connection to be established. |
+| GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY | Server requests client certificate and enforces that the client presents a certificate.<br>If the client presents a certificate, the client authentication is left to the application (the necessary metadata will be available to the application via authentication context properties, see grpc_auth_context).<br>The client's key certificate pair must be valid for the SSL connection to be established. |
+| GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY | Server requests client certificate and enforces that the client presents a certificate.<br>The certificate presented by the client is verified by the gRPC framework. (For a successful connection the client needs to present a certificate that can be verified against the root certificate configured by the server)<br>The client's key certificate pair must be valid for the SSL connection to be established. |
+
+## Initialization with TLS
+
+The desired value is passed as an argument by the [init script](./agent/scripts/openolt) to the **openolt** application.
+
+If the TLS option is an empty string the gRPC server is insecure, otherwise the corresponding TLS level is set.
+
+## Certificate Files
+
+TLS requires valid certificates.
+
+The following certificate files must be present at the **keystore/** directory:
+- root.crt (CA public key),
+- server.crt (public key),
+- server.key (private key)
+
+Self-signed certificates can be used.
+
+## Self-Signed Certificates
+
+Self-signed certificates can be generated by the ubiquitous [OpenSSL](https://en.wikipedia.org/wiki/OpenSSL).
+
+Note that the gRPC clients may require these certificates.
+
+First create the **keystore** directory, if it is not created already.
+A few definitions will be useful, here domain is the ma1 IP address of the OLT.
+
+```
+## on the test OLT
+
+# openssl version
+OpenSSL 1.0.1t  3 May 2016
+
+# mkdir -p /broadcom/keystore
+
+# cd /broadcom/keystore
+
+# DOMAIN=192.168.31.251
+# COMMON_NAME=$DOMAIN
+# SUBJECT="/C=CA/ST=None/L=NB/O=None/CN=$COMMON_NAME"
+# NUM_OF_DAYS=365
+```
+Generate the root certificates:
+
+```
+# openssl genrsa -out root.key 2048
+Generating RSA private key, 2048 bit long modulus
+........................................+++
+..................................+++
+e is 65537 (0x10001)
+root@EC2047000518:~/keystore2#
+root@EC2047000518:~/keystore2# openssl req -x509 -new -nodes -key root.key -sha256 -days $NUM_OF_DAYS -out root.crt
+You are about to be asked to enter information that will be incorporated
+into your certificate request.
+What you are about to enter is what is called a Distinguished Name or a DN.
+There are quite a few fields but you can leave some blank
+For some fields there will be a default value,
+If you enter '.', the field will be left blank.
+-----
+Country Name (2 letter code) [AU]:US
+State or Province Name (full name) [Some-State]:California
+Locality Name (eg, city) []:Santa Clara
+Organization Name (eg, company) [Internet Widgits Pty Ltd]:Netsia
+Organizational Unit Name (eg, section) []:SEBA
+Common Name (e.g. server FQDN or YOUR name) []:www.netsia.com
+Email Address []:contact@netsia.com
+```
+Generate the extension file:
+
+```
+# cat << EOF > v3.ext
+> authorityKeyIdentifier=keyid,issuer
+> basicConstraints=CA:FALSE
+> keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+> subjectAltName = @alt_names
+>
+> [alt_names]
+> DNS.1 = $DOMAIN
+> IP.1 = $COMMON_NAME
+> EOF
+
+# cat v3.ext
+authorityKeyIdentifier=keyid,issuer
+basicConstraints=CA:FALSE
+keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = 192.168.31.251
+IP.1 = 192.168.31.251
+```
+Generate the server certificates:
+
+```
+# openssl req -new -newkey rsa:2048 -sha256 -nodes -keyout server.key -subj "$SUBJECT" -out server.csr
+Generating a 2048 bit RSA private key
+..........................................+++
+..............................................................+++
+writing new private key to 'server.key'
+-----
+
+# openssl x509 -req -in server.csr -CA root.crt -CAkey root.key -CAcreateserial -out server.crt -days $NUM_OF_DAYS -sha256 -extfile v3.ext
+Signature ok
+subject=/C=CA/ST=None/L=NB/O=None/CN=192.168.31.251
+Getting CA Private Key
+```
+## Test with Browsers
+
+Browsers are quite exacting when it comes to certificates, and gRPC is not designed for browsers.
+Still the certificates may be checked with a common browser.
+
+Assuming that the generated certificates will be used by the init script with the first and least-demanding TLS option, going to the URL https://192.168.31.251:9191/ will result in security warnings. Now the certificates can be checked using the browser itself.
+
+Both Google Chrome and Mozilla Firefox respond in a similar way, since they don't know beforehand the Certificate Authority which issued this certificate (self-signed, unknown entitities), they give strong warnings:
+- Google Chrome: NET::ERR_CERT_AUTHORITY_INVALID
+- Mozilla Firefox: SEC_ERROR_UNKNOWN_ISSUER
diff --git a/VERSION b/VERSION
index 47b322c..4d9d11c 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-3.4.1
+3.4.2
diff --git a/agent/common/main.cc b/agent/common/main.cc
index b00b9b5..549bd59 100644
--- a/agent/common/main.cc
+++ b/agent/common/main.cc
@@ -88,7 +88,7 @@
         return 1;
     }
 
-    // Wait for successful activation before allowing VOLTHA to connect. 
+    // Wait for successful activation before allowing VOLTHA to connect.
     // This is necessary to allow the device topology to be dynamically
     // queried from driver after initialization and activation is complete.
     int maxTrials = 300;
@@ -107,7 +107,7 @@
     }
 
     sleep(2);
-    // Enable all PON interfaces. 
+    // Enable all PON interfaces.
     for (int i = 0; i < NumPonIf_(); i++) {
         status = EnablePonIf_(i);
         if (!status.ok()) {
@@ -145,7 +145,11 @@
           break;
        }
     }
-    RunServer(argc, argv);
+
+    if (!RunServer(argc, argv)) {
+        std::cerr << "FATAL: gRPC server creation failed\n";
+        return 2;
+    }
 
     return 0;
 }
diff --git a/agent/common/server.cc b/agent/common/server.cc
index 2b7a915..68cce10 100644
--- a/agent/common/server.cc
+++ b/agent/common/server.cc
@@ -27,6 +27,7 @@
 #include "server.h"
 #include "core.h"
 #include "state.h"
+#include "../src/core_utils.h"
 
 #include <grpc++/grpc++.h>
 #include <voltha_protos/openolt.grpc.pb.h>
@@ -348,8 +349,11 @@
     }
 };
 
-void RunServer(int argc, char** argv) {
+bool RunServer(int argc, char** argv) {
     std::string ipAddress = "0.0.0.0";
+    bool tls_enabled = false;
+    std::pair<grpc_ssl_client_certificate_request_type, bool> grpc_security;
+    std::shared_ptr<grpc::ServerCredentials> credentials;
 
     for (int i = 1; i < argc; ++i) {
         if(strcmp(argv[i-1], "--interface") == 0 || (strcmp(argv[i-1], "--intf") == 0)) {
@@ -358,6 +362,47 @@
         }
     }
 
+    for (int i = 1; i < argc; ++i) {
+        if (strcmp(argv[i-1], "--enable-tls") == 0) {
+            grpc_security = get_grpc_tls_option(argv[i]);
+            if (grpc_security.second) {
+                tls_enabled = true;
+            } else {
+                std::cerr << "unknown security option: \"" << argv[i-1] << " " << argv[i] << "\"\n";
+                return false;
+            };
+            break;
+        }
+    }
+
+    if (tls_enabled) {
+        std::string dir_cert{"./keystore"};
+        auto read_root_crt = read_from_txt_file(dir_cert + "/root.crt");
+        auto read_server_key = read_from_txt_file(dir_cert + "/server.key");
+        auto read_server_crt = read_from_txt_file(dir_cert + "/server.crt");
+
+        if (read_root_crt.second && read_server_key.second && read_server_crt.second) {
+            std::cout << "certificate files read successfully\n";
+        } else {
+            std::cerr << std::boolalpha << "certificate files read failed - root.crt: " << read_root_crt.second
+                                                                    << ", server.key: " << read_server_key.second
+                                                                    << ", server.crt: " << read_server_crt.second << '\n';
+            return false;
+        }
+
+        std::string root_crt = read_root_crt.first;
+        std::string server_key = read_server_key.first;
+        std::string server_crt = read_server_crt.first;
+
+        grpc::SslServerCredentialsOptions ssl_opts{grpc_security.first};
+        ssl_opts.pem_root_certs = root_crt;
+        grpc::SslServerCredentialsOptions::PemKeyCertPair keycert = {server_key, server_crt};
+        ssl_opts.pem_key_cert_pairs.push_back(keycert);
+        credentials = grpc::SslServerCredentials(ssl_opts);
+    } else {
+        credentials = grpc::InsecureServerCredentials();
+    }
+
     serverPort = ipAddress.append(":9191").c_str();
     OpenoltService service;
     std::string server_address(serverPort);
@@ -365,7 +410,7 @@
     ::ResourceQuota quota;
     quota.SetMaxThreads(GRPC_THREAD_POOL_SIZE);
     builder.SetResourceQuota(quota);
-    builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
+    builder.AddListeningPort(server_address, credentials);
     builder.RegisterService(&service);
 
     std::unique_ptr<Server> server(builder.BuildAndStart());
@@ -377,5 +422,11 @@
     std::cout << "Server listening on " << server_address
     << ", connection signature : " << signature << std::endl;
 
+#ifdef TEST_MODE
+    server->Shutdown();
+#else
     server->Wait();
+#endif
+
+    return true;
 }
diff --git a/agent/common/server.h b/agent/common/server.h
index f196fa3..56b8916 100644
--- a/agent/common/server.h
+++ b/agent/common/server.h
@@ -17,6 +17,6 @@
 #ifndef OPENOLT_SERVER_H_
 #define OPENOLT_SERVER_H_
 
-void RunServer(int argc, char** argv);
+bool RunServer(int argc, char** argv);
 
 #endif
diff --git a/agent/device/asfvolt16/mkdebian/debian/asfvolt16.postinst b/agent/device/asfvolt16/mkdebian/debian/asfvolt16.postinst
index b7ae471..ff4b68a 100644
--- a/agent/device/asfvolt16/mkdebian/debian/asfvolt16.postinst
+++ b/agent/device/asfvolt16/mkdebian/debian/asfvolt16.postinst
@@ -37,3 +37,5 @@
 /opt/bcm68620/svk_init.sh
 [ "A`cat ~/.bashrc| grep LD_LIBRARY_PATH`" = "A" ] && echo "export LD_LIBRARY_PATH=/broadcom:\$LD_LIBRARY_PATH" >> ~/.bashrc; . ~/.bashrc
 ldconfig /broadcom
+# create directory for certificates
+mkdir -p /broadcom/keystore/
diff --git a/agent/device/asgvolt64/mkdebian/debian/asgvolt64.postinst b/agent/device/asgvolt64/mkdebian/debian/asgvolt64.postinst
index c930f7b..46e2356 100644
--- a/agent/device/asgvolt64/mkdebian/debian/asgvolt64.postinst
+++ b/agent/device/asgvolt64/mkdebian/debian/asgvolt64.postinst
@@ -37,3 +37,5 @@
 /opt/bcm68620/svk_init.sh
 [ "A`cat ~/.bashrc| grep LD_LIBRARY_PATH`" = "A" ] && echo "export LD_LIBRARY_PATH=/broadcom:\$LD_LIBRARY_PATH" >> ~/.bashrc; . ~/.bashrc
 ldconfig /broadcom
+# create directory for certificates
+mkdir -p /broadcom/keystore/
diff --git a/agent/scripts/init.d/openolt b/agent/scripts/init.d/openolt
index e9a448b..27dfa0c 100755
--- a/agent/scripts/init.d/openolt
+++ b/agent/scripts/init.d/openolt
@@ -19,11 +19,44 @@
 USER="root"
 GROUP="root"
 
+# ------------------------------------------------------------------------------
 # If OLT is used in inband mode, inband interface name will be copied
 # to /etc/default/openolt file. Here inband interface is passed as argument
 # while running openolt service
 [ -r /etc/default/openolt ] && . /etc/default/openolt
 [ -z "gRPC_interface" ] || APPARGS="--interface $gRPC_interface"
+# ------------------------------------------------------------------------------
+GRPC_TLS_OPTION_A='GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE'
+# Server does not request client certificate.
+# The certificate presented by the client is not checked by the server at all. (A client may present a self signed or signed certificate or not present a certificate at all and any of those option would be accepted)
+
+GRPC_TLS_OPTION_B='GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY'
+# Server requests client certificate but does not enforce that the client presents a certificate.
+# If the client presents a certificate, the client authentication is left to the application (the necessary metadata will be available to the application via authentication context properties, see grpc_auth_context).
+# The client's key certificate pair must be valid for the SSL connection to be established.
+
+GRPC_TLS_OPTION_C='GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY'
+# Server requests client certificate but does not enforce that the client presents a certificate.
+# If the client presents a certificate, the client authentication is done by the gRPC framework. (For a successful connection the client needs to either present a certificate that can be verified against the root certificate configured by the server or not present a certificate at all)
+# The client's key certificate pair must be valid for the SSL connection to be established.
+
+GRPC_TLS_OPTION_D='GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY'
+# Server requests client certificate and enforces that the client presents a certificate.
+# If the client presents a certificate, the client authentication is left to the application (the necessary metadata will be available to the application via authentication context properties, see grpc_auth_context).
+# The client's key certificate pair must be valid for the SSL connection to be established.
+
+GRPC_TLS_OPTION_E='GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY'
+# Server requests client certificate and enforces that the client presents a certificate.
+# The certificate presented by the client is verified by the gRPC framework. (For a successful connection the client needs to present a certificate that can be verified against the root certificate configured by the server)
+# The client's key certificate pair must be valid for the SSL connection to be established.
+
+GRPC_TLS_OPTION_Z='' # INSECURE
+
+# choose one of the above six options with the last letter
+GRPC_TLS_OPTION=$GRPC_TLS_OPTION_Z
+
+[ $GRPC_TLS_OPTION ] && APPARGS="$APPARGS --enable-tls $GRPC_TLS_OPTION"
+# ------------------------------------------------------------------------------
 
 # Include functions
 set -e
@@ -52,7 +85,7 @@
   while test -d /proc/$(cat /var/run/$NAME.pid); do
     killtree $(cat /var/run/$NAME.pid) 15
     sleep 0.5
-  done 
+  done
   [ -z `cat /var/run/$NAME.pid 2>/dev/null` ] || rm /var/run/$NAME.pid
   printf "done\n"
 }
diff --git a/agent/src/core_utils.cc b/agent/src/core_utils.cc
index 59c7cdc..0153078 100644
--- a/agent/src/core_utils.cc
+++ b/agent/src/core_utils.cc
@@ -14,8 +14,13 @@
  * limitations under the License.
  */
 
+#include <fstream>
+#include <sstream>
 #include "core_utils.h"
 
+// save the TLS option
+static std::string tls_option_arg{};
+
 std::string serial_number_to_str(bcmolt_serial_number* serial_number) {
 #define SERIAL_NUMBER_SIZE 12
     char buff[SERIAL_NUMBER_SIZE+1];
@@ -1673,3 +1678,48 @@
     }
     return false;
 }
+
+std::pair<grpc_ssl_client_certificate_request_type, bool> get_grpc_tls_option(const char* tls_option) {
+    static std::map<std::string,grpc_ssl_client_certificate_request_type> grpc_security_option_map = {{"GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE",
+                                                                                                        grpc_ssl_client_certificate_request_type::GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE},
+                                                                                                      {"GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY",
+                                                                                                        grpc_ssl_client_certificate_request_type::GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY},
+                                                                                                      {"GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY",
+                                                                                                        grpc_ssl_client_certificate_request_type::GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY},
+                                                                                                      {"GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY",
+                                                                                                        grpc_ssl_client_certificate_request_type::GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY},
+                                                                                                      {"GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY",
+                                                                                                        grpc_ssl_client_certificate_request_type::GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY}};
+
+    auto it = grpc_security_option_map.find(tls_option);
+    if (it == grpc_security_option_map.end()) {
+        OPENOLT_LOG(ERROR, openolt_log_id, "invalid gRPC Server security option: %s\n", tls_option);
+        return {grpc_ssl_client_certificate_request_type::GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE, false};
+    } else {
+        OPENOLT_LOG(INFO, openolt_log_id, "valid gRPC Server security option: %s\n", tls_option);
+        tls_option_arg = std::string{tls_option};
+        return {it->second, true};
+    }
+}
+
+const std::string &get_grpc_tls_option() {
+    return tls_option_arg;
+}
+
+bool is_grpc_secure() {
+    return !tls_option_arg.empty();
+}
+
+std::pair<std::string, bool> read_from_txt_file(const std::string& file_name) {
+    std::ifstream in_file(file_name);
+
+    if (!in_file.is_open()) {
+        OPENOLT_LOG(ERROR, openolt_log_id, "error opening file '%s'\n", file_name.c_str());
+        return {"", false};
+    }
+
+    std::stringstream buffer;
+    buffer << in_file.rdbuf();
+
+    return {buffer.str(), in_file.good()};
+}
diff --git a/agent/src/core_utils.h b/agent/src/core_utils.h
index 0f7cfe7..3d4668c 100644
--- a/agent/src/core_utils.h
+++ b/agent/src/core_utils.h
@@ -25,6 +25,7 @@
 #include "core.h"
 #include "core_data.h"
 #include "error_format.h"
+#include <grpc/grpc_security_constants.h>
 
 extern "C"
 {
@@ -117,4 +118,8 @@
 const device_flow_params* get_device_flow_params(uint64_t voltha_flow_id);
 trap_to_host_packet_type get_trap_to_host_packet_type(const ::openolt::Classifier& classifier);
 bool is_packet_allowed(bcmolt_access_control_receive_eth_packet_data *data, int32_t gemport_id);
+std::pair<grpc_ssl_client_certificate_request_type, bool> get_grpc_tls_option(const char* tls_option);
+const std::string &get_grpc_tls_option();
+bool is_grpc_secure();
+std::pair<std::string, bool> read_from_txt_file(const std::string& file_name);
 #endif // OPENOLT_CORE_UTILS_H_
diff --git a/agent/test/keystore-test/root.crt b/agent/test/keystore-test/root.crt
new file mode 100644
index 0000000..2ae357b
--- /dev/null
+++ b/agent/test/keystore-test/root.crt
@@ -0,0 +1,24 @@
+-----BEGIN CERTIFICATE-----
+MIIEDTCCAvWgAwIBAgIJALTkUm+wi0Y9MA0GCSqGSIb3DQEBCwUAMIGcMQswCQYD
+VQQGEwJUUjERMA8GA1UECAwISXN0YW5idWwxETAPBgNVBAcMCElzdGFuYnVsMQ8w
+DQYDVQQKDAZBUkdFTEExDTALBgNVBAsMBFNFQkExGzAZBgNVBAMMEnNlYmEuYXJn
+ZWxhLmNvbS50cjEqMCgGCSqGSIb3DQEJARYbbmV0c2lhLmRldm9wc0BhcmdlbGEu
+Y29tLnRyMB4XDTIxMDQxMzIzMTY1OFoXDTMxMDQxMTIzMTY1OFowgZwxCzAJBgNV
+BAYTAlRSMREwDwYDVQQIDAhJc3RhbmJ1bDERMA8GA1UEBwwISXN0YW5idWwxDzAN
+BgNVBAoMBkFSR0VMQTENMAsGA1UECwwEU0VCQTEbMBkGA1UEAwwSc2ViYS5hcmdl
+bGEuY29tLnRyMSowKAYJKoZIhvcNAQkBFhtuZXRzaWEuZGV2b3BzQGFyZ2VsYS5j
+b20udHIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCwos2j7GWNExJZ
+PgeOURaP/9JcV+1SC5aDqG4UwLlAGSPoQUkf/0LjgMde9ZeRe8rugGXW39v2cbWm
+eb5TNyTwWGED2wEMVijzBgnYTYA6nXTxz+XAvxVu/x6fHtNMOqHGiewN0S1eY7bz
+rE1zq6SeunDJGCW6pIa4aNMq7/ngDsPrSc6f1pKZpCgrNSW/LnGdgTB2HkaroUr8
+vy3OWZ0FG042HjT7q+MjEgqGLDl7VkBFrJn2gp4sfgX1WSM/9brhcaVZ/puagaY8
+Ju2Dq3foXDW3xgW9y22mk/3cQpRGRSad9+ZAWb9ZPyp5lcUaqpMuYyiFoGctdQT8
+s1LMbPCnAgMBAAGjUDBOMB0GA1UdDgQWBBTpfNEjdHBqnzSDYXuwrFCnZ7P1/zAf
+BgNVHSMEGDAWgBTpfNEjdHBqnzSDYXuwrFCnZ7P1/zAMBgNVHRMEBTADAQH/MA0G
+CSqGSIb3DQEBCwUAA4IBAQBorTfS9g2xi9kO42nDTs+VRv64j09cJyZl5BFoivFO
+SV2O64ubp6YEaKjqUnPuzs7D3j8cjalqeIeD2tFJHRrv8XADlgivEtTk2o5sCEJx
+66eOAXeqlLPHK9dt3IxKI5kPBTGKQHw2uNG8VOsRlq2DB798ueWxzUWPpxjhSFL/
+YP/o3pEWxit5SirBAoaRZaccjPHz2c/4IfGuoPOXun+qKvt/ic2QRm23e62Wb/Do
+eugVy1Zi0aAOenoJaNuiZEX7l2qZ4RssGheJxWdEOzGZ9YxpS7WseKgxufQ9RmoR
+4J4bU/hVy2su6+cqn7Q5YchcJjfuT181yq+/UFgpvMTa
+-----END CERTIFICATE-----
diff --git a/agent/test/keystore-test/server.crt b/agent/test/keystore-test/server.crt
new file mode 100644
index 0000000..8186f31
--- /dev/null
+++ b/agent/test/keystore-test/server.crt
@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIIDzTCCArWgAwIBAgIJAILLZCQ2zGU7MA0GCSqGSIb3DQEBCwUAMIGcMQswCQYD
+VQQGEwJUUjERMA8GA1UECAwISXN0YW5idWwxETAPBgNVBAcMCElzdGFuYnVsMQ8w
+DQYDVQQKDAZBUkdFTEExDTALBgNVBAsMBFNFQkExGzAZBgNVBAMMEnNlYmEuYXJn
+ZWxhLmNvbS50cjEqMCgGCSqGSIb3DQEJARYbbmV0c2lhLmRldm9wc0BhcmdlbGEu
+Y29tLnRyMB4XDTIxMDQxMzIzMTcyNFoXDTMxMDQxMTIzMTcyNFowUTELMAkGA1UE
+BhMCQ0ExDTALBgNVBAgMBE5vbmUxCzAJBgNVBAcMAk5CMQ0wCwYDVQQKDAROb25l
+MRcwFQYDVQQDDA4xOTIuMTY4LjMwLjIyMTCCASIwDQYJKoZIhvcNAQEBBQADggEP
+ADCCAQoCggEBANh3OVBIS7kFcOda1dA8+423RRhfrwsL3AHjthZ6+f9TYSb0BcFv
+VQKwGRWXVFZsX3q6yivi8zq1AoLUYd7Ir0W+UFuP17unMeKij3PjPMwc42aL5lfs
+7b1VVG1yk4CoMHdGs/gpbLQwAvhwvaovI1SReCW8ddehJ6P50P8ACVmklK08W9Hm
+mdzAdq/HUw/oaCi5kNtFP2aqXlOofYaJ6aYRLaGYNpgvj9SiGo62SB0VRteik4Pd
+HjjiTkWUcEblUC89UfQCcqGUG4vXoNhaS2AmMCg+MgPlaWRyUT6q9uXQ5CnLSD9r
+lDcSNsHHwapMf1lWPBPB2r0EkQpbazD+WWsCAwEAAaNcMFowHwYDVR0jBBgwFoAU
+6XzRI3Rwap80g2F7sKxQp2ez9f8wCQYDVR0TBAIwADALBgNVHQ8EBAMCBPAwHwYD
+VR0RBBgwFoIOMTkyLjE2OC4zMC4yMjGHBMCoHt0wDQYJKoZIhvcNAQELBQADggEB
+AIHUdvm/njSX5wQzegQeq7Axo+o6HfhJFH5ot1B0rtLG3/SkWOj+t9gqEeJxLYCI
+Wz+EhUupISZ/Vb8KtSZgLPFQ+H61+x945zT0zE61Wh3hKJOK1HZVbMhtWHbzaqEa
+nPQniw9wCKflnUgJUqYGb3rBFqmCEKYXvcyQp63zUmnAIJze+liEXVmHpCPPR+Jo
+CcoSXMR7iCl6s1hpkSB72CxhvcZi6zfCkyVTmxDfwVTtciqjQEr8gC2ufff7qRHM
+ZRrU9NtiMQqu5yZQsQiFGMDU68sHYz3uQQSjaqih+DcPLf7Pg1qFrrbbq7C0Znmw
+rCp11vZTwWKj3fg/lR7AAq4=
+-----END CERTIFICATE-----
diff --git a/agent/test/keystore-test/server.key b/agent/test/keystore-test/server.key
new file mode 100644
index 0000000..371ceeb
--- /dev/null
+++ b/agent/test/keystore-test/server.key
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDYdzlQSEu5BXDn
+WtXQPPuNt0UYX68LC9wB47YWevn/U2Em9AXBb1UCsBkVl1RWbF96usor4vM6tQKC
+1GHeyK9FvlBbj9e7pzHioo9z4zzMHONmi+ZX7O29VVRtcpOAqDB3RrP4KWy0MAL4
+cL2qLyNUkXglvHXXoSej+dD/AAlZpJStPFvR5pncwHavx1MP6GgouZDbRT9mql5T
+qH2GiemmES2hmDaYL4/UohqOtkgdFUbXopOD3R444k5FlHBG5VAvPVH0AnKhlBuL
+16DYWktgJjAoPjID5WlkclE+qvbl0OQpy0g/a5Q3EjbBx8GqTH9ZVjwTwdq9BJEK
+W2sw/llrAgMBAAECggEAJPrZM7nmFcm0LBH8jQKeBuvURgGyqbRw9XGtXdHMe64G
+3ty5Tv5yC2+CGYoswkbpWNIoT68J1nGt6CPYRHMMXmHQ4JOGS2yYvRsAXs/nRwd4
+YcBsrYCwbLPzJG96e3RAM2BExpRtGatKFp9MB5Ld+kiPugKkLx4842p+JdIfxsxi
+lggzGkneu95yJnTS2i/BcKUAVO2HM1DKOtfgIofPCT4tgZpboa0p7GKa1TtqqdTT
+Zk4t6qVh3MSRYcQe2pMXZJgOAJupAzXIAYoafw+io22dt2mDqLhb9YRxCIabhYif
+Oz8e4b9O8hvJcTs0/2hYUW4vtDQzlFCeSaC5fuk5oQKBgQDw04BnrkQ84kPZ0PrN
+e8zWk2jAqBmiyzgykMbUnX34CImINTjwgqhgKjMFepqxGwsQN7YHyu5Jq7I1sEYm
+LSrQGDa8zqTifcSkyHq7k+TFM371/OCW+xAUITrzd/KZdBiH2syAePHUO/02G7up
+aIUjvaB2Qp/qVPe3MzDkVGvDNQKBgQDmGsp4VdL/SEsWCSs+5tIK2gnok2+/V9s5
+fSPPROABsEBtLf+rJeGmh6qkOBeRZnncAajza6oKMKo9NLA68qFNs7Yg+ZvnDcu7
+X+VyRiXp+uR65oNPEhIMO8x7NGPx+VqYZtAWI0I3lGZpzkZ8zfVHFJCFpd+ieDv8
+RSAHX3ueHwKBgGjSxrtaYR1XJ01x5xj14A0uEJR7Ft2DMyzfU4xMGP6TSLne4K/f
+T0a8V4T0/sxEBybB+RufOuUaNPGljoMjedUmFP1NvPtQ3v0SvklTcGpdpc+QiHlv
+QmpgpHZBKXmdzOelVwo8mpZUnjylCaQNeJY7/dI74btvzHsTbx6TmGxZAoGBAOER
+3W8cy3hl1wPjzggFr/drU/vIkqo/HjA6Jherj8w3AJ2KO6TFNdU0qAVe1DalXJaE
+jSQj8DttZGbfrWzLPFmLaZ8RZ5v104wgfYZr9NPLU0afSFrEGyEaMKVmFkhtlV6y
+WeD9ddx1bEMbv7h9n+d5xu7i0z7QiiPz3SM5EuoTAoGBAJKBvQx7ZtW2QMgdwxdF
+2Yfv7qmifUcsi7vLNeEQGSBa6s7SXPLrUaODSIKL0lIgIsxQelxJj8zfDkqVEWqO
+8b0g+v9FAqQjgB90mALPNcdS/DCuQ2Vdx0ertkGjUetWaAhuHspZi79IOG+2j9iv
+pvZKgzwl2RaJTlr77Qz6lsa4
+-----END PRIVATE KEY-----
diff --git a/agent/test/src/test_core.cc b/agent/test/src/test_core.cc
index bce275e..55297b9 100644
--- a/agent/test/src/test_core.cc
+++ b/agent/test/src/test_core.cc
@@ -18,6 +18,7 @@
 #include "bal_mocker.h"
 #include "core.h"
 #include "core_data.h"
+#include "server.h"
 #include <future>
 using namespace testing;
 using namespace std;
@@ -3239,3 +3240,87 @@
     Status status = GetLogicalOnuDistance_(pon_ni, onu_id, onu_logical_distance);
     ASSERT_TRUE( status.error_message() != Status::OK.error_message() );
 }
+
+////////////////////////////////////////////////////////////////////////////
+// For testing Secure Server functionality
+////////////////////////////////////////////////////////////////////////////
+
+class TestSecureServer : public Test {
+    protected:
+        virtual void SetUp() {}
+        virtual void TearDown() {}
+};
+
+TEST_F(TestSecureServer, StartInsecureServer) {
+    // const to prevent the following warning:
+    // warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
+    const char *args[] = {"./openolt"};
+    int argc = sizeof(args) / sizeof(args[0]);
+    char **argv = const_cast<char**>(args);
+
+    bool ok = RunServer(argc, argv);
+
+    ASSERT_TRUE(ok);
+
+    OPENOLT_LOG(INFO, openolt_log_id, "insecure gRPC server has been started and shut down successfully\n");
+}
+
+TEST_F(TestSecureServer, StartWithInvalidTLSOption) {
+    const char *args[] = {"./openolt", "--enable-tls", "DUMMY_GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE"};
+    int argc = sizeof(args) / sizeof(args[0]);
+    char **argv = const_cast<char**>(args);
+
+    bool ok = RunServer(argc, argv);
+
+    ASSERT_FALSE(ok);
+
+    OPENOLT_LOG(INFO, openolt_log_id, "secure gRPC server could not be started due to invalid TLS option\n");
+}
+
+TEST_F(TestSecureServer, CertificatesAreMissing) {
+    const char *args[] = {"./openolt", "--enable-tls", "GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE"};
+    int argc = sizeof(args) / sizeof(args[0]);
+    char **argv = const_cast<char**>(args);
+    const std::string cmd = "exec [ -d './keystore' ] && rm -rf './keystore'";
+
+    int res = std::system(cmd.c_str());
+    if (res == 0) {
+        std::cout << "directory ./keystore is deleted\n";
+    } else {
+        std::cout << "directory ./keystore is not existing\n";
+    }
+
+    bool ok = RunServer(argc, argv);
+
+    ASSERT_FALSE(ok);
+
+    OPENOLT_LOG(INFO, openolt_log_id, "secure gRPC server could not be started due to missing certificates\n");
+}
+
+TEST_F(TestSecureServer, StartWithValidTLSOption) {
+    const char *args[] = {"./openolt", "--enable-tls", "GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE"};
+    int argc = sizeof(args) / sizeof(args[0]);
+    char **argv = const_cast<char**>(args);
+    const std::string cmd_1 = "exec cp -r ./keystore-test ./keystore";
+    const std::string cmd_2 = "exec rm -rf './keystore'";
+
+    int res = std::system(cmd_1.c_str());
+    if (res == 0) {
+        std::cout << "directory ./keystore is copied from ./keystore-test\n";
+
+        bool ok = RunServer(argc, argv);
+        ASSERT_TRUE(ok);
+
+        OPENOLT_LOG(INFO, openolt_log_id, "secure gRPC server has been started with the given certificates and TLS options, and shut down successfully\n");
+
+        res = std::system(cmd_2.c_str());
+        if (res == 0) {
+            std::cout << "directory ./keystore is deleted\n";
+        } else {
+            std::cerr << "directory ./keystore could not be deleted\n";
+        }
+    } else {
+        std::cerr << "directory ./keystore could not be prepared, err: " << res << '\n';
+        FAIL();
+    }
+}