blob: a52a12055b9ca7d3e8eff38b98aed86d2e6ad7f6 [file] [log] [blame]
KafkaLogHandler
===============
Provides a python ``logging`` compatible handler for producing messages to a
Kafka message bus.
Depends on the confluent_kafka module to connect to Kafka
Designed to support structured logging, and serializes log data as JSON when
published as a Kafka message.
Usage
=====
**Example:**
::
import logger
from kafkaloghandler import KafkaLogHandler
log = logging.getLogger()
klh = KafkaLogHandler(bootstrap_servers=["test-kafka:9092"], topic="testtopic")
log.addHandler(klh)
data={'example':'structured data'}
log.info('message to send to kafka', data=data)
**Parameters that can be provided to KafkaLogHandler:**
*bootstrap_servers*
List of Kafka bootstrap servers to connect to. See confluent_kafka docs.
**default:** ``["localhost:9092"]``
*timeout*
Timeout in seconds for flushing producer queue. See confluent_kafka docs.
**default:** ``10.0``
*topic*
String that sets the topic in Kafka.
**default:** ``"kafkaloghandler"``
*key*
String that sets the default key in Kafka, can be used for summarization within Kafka.
NOTE: This default key can be overridden on a per-message basis by passing a
dict to the logger with ``{"key": "new_key_for_this_message"}`` in it.
**default:** ``"klh"``
*flatten*
Flattens nested dictionary keys passed as structured logging into the parent
dictionary layer, up to a certain depth.
This is useful when logging to external systems that don't have good support
for hierarchical data.
Example: ``{'a': {'b': 'c'}}`` would be flattened to ``{'a.b': 'c'}``
If the depth is exceeded, any remaining deeper dict will be added to the
output under the flattened key.
Set to ``0`` to turn off flattening.
**default:** ``5``
*separator*
Separator used between keys when flattening.
**default:** ``.``
*blacklist*
List of top-level keys to discard from structured logs when outputting JSON.
**default:** ``['_logger']``
Tests
=====
Unit tests can be run with:
nose2 --verbose --coverage-report term