VOL-1183: OFagent cannot reach vcore after vcore moved to different node in cluster
Two gRPC calls made by OFagent's ConnectionManager, Subscribe and ListLogicalDevices, actually
run in the Twisted reactor thread. On rare occasions, either call never returns when the vcore
connected to the OFagent dies. Because the blocking gRPC runs in the reactor thread, the entire
application is blocked and cannot detect comms errors with the deceased vcore. The handling of
these gRPCs has been moved to the OFagent's internal GrpcClient, which wraps all gRPC calls in
the twisted.internet.threads.deferToThread function.
Change-Id: Ifa3479a3cbca7ae8d7f5e2ca0372e22507d5e4b9
diff --git a/ofagent/main.py b/ofagent/main.py
index 23335bb..06c2ae3 100755
--- a/ofagent/main.py
+++ b/ofagent/main.py
@@ -203,6 +203,9 @@
self.args = args = parse_args()
self.config = load_config(args)
+ # May want to specify the gRPC timeout as an arg (in future)
+ # Right now, set a default value
+ self.grpc_timeout = 10
verbosity_adjust = (args.verbose or 0) - (args.quiet or 0)
self.log = setup_logging(self.config.get('logging', {}),
@@ -227,7 +230,8 @@
self.log.info('starting-internal-components')
args = self.args
self.connection_manager = yield ConnectionManager(
- args.consul, args.grpc_endpoint, args.controller, args.instance_id, \
+ args.consul, args.grpc_endpoint, self.grpc_timeout,
+ args.controller, args.instance_id,
args.enable_tls, args.key_file, args.cert_file).start()
self.log.info('started-internal-services')