Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 1 | KafkaLogHandler |
| 2 | =============== |
| 3 | |
| 4 | Provides a python ``logging`` compatible handler for producing messages to a |
| 5 | Kafka message bus. |
| 6 | |
Zack Williams | 1f30002 | 2018-10-26 15:30:23 -0700 | [diff] [blame] | 7 | Depends on the confluent_kafka module to connect to Kafka. |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 8 | |
Zack Williams | 1f30002 | 2018-10-26 15:30:23 -0700 | [diff] [blame] | 9 | Designed to support both standard and structlog formats, and serializes log |
| 10 | data as JSON when published as a Kafka message. Messages are normalized to be |
| 11 | more compatible with Logstash/Filebeat formats. |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 12 | |
| 13 | Usage |
| 14 | ===== |
| 15 | |
| 16 | **Example:** |
| 17 | |
| 18 | :: |
| 19 | |
| 20 | import logger |
| 21 | |
| 22 | from kafkaloghandler import KafkaLogHandler |
| 23 | |
| 24 | log = logging.getLogger() |
| 25 | |
| 26 | klh = KafkaLogHandler(bootstrap_servers=["test-kafka:9092"], topic="testtopic") |
| 27 | |
| 28 | log.addHandler(klh) |
| 29 | |
| 30 | data={'example':'structured data'} |
| 31 | |
| 32 | log.info('message to send to kafka', data=data) |
| 33 | |
| 34 | |
| 35 | **Parameters that can be provided to KafkaLogHandler:** |
| 36 | |
| 37 | *bootstrap_servers* |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 38 | List of Kafka bootstrap servers to connect to. |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 39 | |
| 40 | **default:** ``["localhost:9092"]`` |
| 41 | |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 42 | *extra_config* |
| 43 | Dictionary of extra `producer configuration |
| 44 | <https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md>`_ |
| 45 | passed to librdkafka. |
| 46 | |
| 47 | NOTE: The ``bootstrap_servers`` parameter will overwrite |
| 48 | ``bootstrap.servers``. |
| 49 | |
| 50 | **default:** ``{}`` |
| 51 | |
| 52 | |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 53 | *timeout* |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 54 | Timeout in seconds for flushing producer queue. See librdkafka docs. |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 55 | |
| 56 | **default:** ``10.0`` |
| 57 | |
| 58 | *topic* |
| 59 | String that sets the topic in Kafka. |
| 60 | |
| 61 | **default:** ``"kafkaloghandler"`` |
| 62 | |
| 63 | *key* |
| 64 | String that sets the default key in Kafka, can be used for summarization within Kafka. |
| 65 | |
| 66 | NOTE: This default key can be overridden on a per-message basis by passing a |
| 67 | dict to the logger with ``{"key": "new_key_for_this_message"}`` in it. |
| 68 | |
| 69 | **default:** ``"klh"`` |
| 70 | |
| 71 | *flatten* |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 72 | Flattens nested dictionaries and lists passed as structured logging into the parent |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 73 | dictionary layer, up to a certain depth. |
| 74 | |
| 75 | This is useful when logging to external systems that don't have good support |
| 76 | for hierarchical data. |
| 77 | |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 78 | Example dictionary: ``{'a': {'b': 'c'}}`` would be flattened to ``{'a.b': 'c'}`` |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 79 | |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 80 | Example list: ``{'a': ['b', 'c']}`` would be flattened to ``{'a.0': 'b', 'a.1': 'c'}`` |
| 81 | |
| 82 | If the depth is exceeded, any remaining deeper items will be added to the |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 83 | output under the flattened key. |
| 84 | |
| 85 | Set to ``0`` to turn off flattening. |
| 86 | |
| 87 | **default:** ``5`` |
| 88 | |
| 89 | *separator* |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 90 | Separator used between items when flattening. |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 91 | |
| 92 | **default:** ``.`` |
| 93 | |
| 94 | *blacklist* |
| 95 | List of top-level keys to discard from structured logs when outputting JSON. |
| 96 | |
Zack Williams | 1f30002 | 2018-10-26 15:30:23 -0700 | [diff] [blame] | 97 | **default:** ``["_logger", "_name"]`` |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 98 | |
| 99 | |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 100 | Testing |
| 101 | ======= |
Zack Williams | 9e8efd3 | 2018-10-17 15:01:13 -0700 | [diff] [blame] | 102 | |
Zack Williams | 19e1cc2 | 2018-11-05 23:18:41 -0700 | [diff] [blame] | 103 | Unit tests can be run with ``tox`` |