kesavand | 2cde658 | 2020-06-22 04:56:23 -0400 | [diff] [blame] | 1 | /* |
| 2 | Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level |
| 3 | API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level |
| 4 | API is insufficient. Usage examples for the high-level APIs are provided inline with their full documentation. |
| 5 | |
| 6 | To produce messages, use either the AsyncProducer or the SyncProducer. The AsyncProducer accepts messages on a channel |
| 7 | and produces them asynchronously in the background as efficiently as possible; it is preferred in most cases. |
| 8 | The SyncProducer provides a method which will block until Kafka acknowledges the message as produced. This can be |
| 9 | useful but comes with two caveats: it will generally be less efficient, and the actual durability guarantees |
| 10 | depend on the configured value of `Producer.RequiredAcks`. There are configurations where a message acknowledged by the |
| 11 | SyncProducer can still sometimes be lost. |
| 12 | |
| 13 | To consume messages, use Consumer or Consumer-Group API. |
| 14 | |
| 15 | For lower-level needs, the Broker and Request/Response objects permit precise control over each connection |
| 16 | and message sent on the wire; the Client provides higher-level metadata management that is shared between |
| 17 | the producers and the consumer. The Request/Response objects and properties are mostly undocumented, as they line up |
| 18 | exactly with the protocol fields documented by Kafka at |
| 19 | https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol |
| 20 | |
| 21 | Metrics are exposed through https://github.com/rcrowley/go-metrics library in a local registry. |
| 22 | |
| 23 | Broker related metrics: |
| 24 | |
| 25 | +----------------------------------------------+------------+---------------------------------------------------------------+ |
| 26 | | Name | Type | Description | |
| 27 | +----------------------------------------------+------------+---------------------------------------------------------------+ |
| 28 | | incoming-byte-rate | meter | Bytes/second read off all brokers | |
| 29 | | incoming-byte-rate-for-broker-<broker-id> | meter | Bytes/second read off a given broker | |
| 30 | | outgoing-byte-rate | meter | Bytes/second written off all brokers | |
| 31 | | outgoing-byte-rate-for-broker-<broker-id> | meter | Bytes/second written off a given broker | |
| 32 | | request-rate | meter | Requests/second sent to all brokers | |
| 33 | | request-rate-for-broker-<broker-id> | meter | Requests/second sent to a given broker | |
| 34 | | request-size | histogram | Distribution of the request size in bytes for all brokers | |
| 35 | | request-size-for-broker-<broker-id> | histogram | Distribution of the request size in bytes for a given broker | |
| 36 | | request-latency-in-ms | histogram | Distribution of the request latency in ms for all brokers | |
| 37 | | request-latency-in-ms-for-broker-<broker-id> | histogram | Distribution of the request latency in ms for a given broker | |
| 38 | | response-rate | meter | Responses/second received from all brokers | |
| 39 | | response-rate-for-broker-<broker-id> | meter | Responses/second received from a given broker | |
| 40 | | response-size | histogram | Distribution of the response size in bytes for all brokers | |
| 41 | | response-size-for-broker-<broker-id> | histogram | Distribution of the response size in bytes for a given broker | |
kesavand | c71914f | 2022-03-25 11:19:03 +0530 | [diff] [blame] | 42 | | requests-in-flight | counter | The current number of in-flight requests awaiting a response | |
| 43 | | | | for all brokers | |
| 44 | | requests-in-flight-for-broker-<broker-id> | counter | The current number of in-flight requests awaiting a response | |
| 45 | | | | for a given broker | |
kesavand | 2cde658 | 2020-06-22 04:56:23 -0400 | [diff] [blame] | 46 | +----------------------------------------------+------------+---------------------------------------------------------------+ |
| 47 | |
| 48 | Note that we do not gather specific metrics for seed brokers but they are part of the "all brokers" metrics. |
| 49 | |
| 50 | Producer related metrics: |
| 51 | |
| 52 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 53 | | Name | Type | Description | |
| 54 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 55 | | batch-size | histogram | Distribution of the number of bytes sent per partition per request for all topics | |
| 56 | | batch-size-for-topic-<topic> | histogram | Distribution of the number of bytes sent per partition per request for a given topic | |
| 57 | | record-send-rate | meter | Records/second sent to all topics | |
| 58 | | record-send-rate-for-topic-<topic> | meter | Records/second sent to a given topic | |
| 59 | | records-per-request | histogram | Distribution of the number of records sent per request for all topics | |
| 60 | | records-per-request-for-topic-<topic> | histogram | Distribution of the number of records sent per request for a given topic | |
| 61 | | compression-ratio | histogram | Distribution of the compression ratio times 100 of record batches for all topics | |
| 62 | | compression-ratio-for-topic-<topic> | histogram | Distribution of the compression ratio times 100 of record batches for a given topic | |
| 63 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 64 | |
| 65 | Consumer related metrics: |
| 66 | |
| 67 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 68 | | Name | Type | Description | |
| 69 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 70 | | consumer-batch-size | histogram | Distribution of the number of messages in a batch | |
kesavand | c71914f | 2022-03-25 11:19:03 +0530 | [diff] [blame] | 71 | | consumer-group-join-total-<GroupID> | counter | Total count of consumer group join attempts | |
| 72 | | consumer-group-join-failed-<GroupID> | counter | Total count of consumer group join failures | |
| 73 | | consumer-group-sync-total-<GroupID> | counter | Total count of consumer group sync attempts | |
| 74 | | consumer-group-sync-failed-<GroupID> | counter | Total count of consumer group sync failures | |
kesavand | 2cde658 | 2020-06-22 04:56:23 -0400 | [diff] [blame] | 75 | +-------------------------------------------+------------+--------------------------------------------------------------------------------------+ |
| 76 | |
| 77 | */ |
| 78 | package sarama |
| 79 | |
| 80 | import ( |
kesavand | c71914f | 2022-03-25 11:19:03 +0530 | [diff] [blame] | 81 | "io" |
kesavand | 2cde658 | 2020-06-22 04:56:23 -0400 | [diff] [blame] | 82 | "log" |
| 83 | ) |
| 84 | |
| 85 | var ( |
| 86 | // Logger is the instance of a StdLogger interface that Sarama writes connection |
| 87 | // management events to. By default it is set to discard all log messages via ioutil.Discard, |
| 88 | // but you can set it to redirect wherever you want. |
kesavand | c71914f | 2022-03-25 11:19:03 +0530 | [diff] [blame] | 89 | Logger StdLogger = log.New(io.Discard, "[Sarama] ", log.LstdFlags) |
kesavand | 2cde658 | 2020-06-22 04:56:23 -0400 | [diff] [blame] | 90 | |
| 91 | // PanicHandler is called for recovering from panics spawned internally to the library (and thus |
| 92 | // not recoverable by the caller's goroutine). Defaults to nil, which means panics are not recovered. |
| 93 | PanicHandler func(interface{}) |
| 94 | |
| 95 | // MaxRequestSize is the maximum size (in bytes) of any request that Sarama will attempt to send. Trying |
| 96 | // to send a request larger than this will result in an PacketEncodingError. The default of 100 MiB is aligned |
| 97 | // with Kafka's default `socket.request.max.bytes`, which is the largest request the broker will attempt |
| 98 | // to process. |
| 99 | MaxRequestSize int32 = 100 * 1024 * 1024 |
| 100 | |
| 101 | // MaxResponseSize is the maximum size (in bytes) of any response that Sarama will attempt to parse. If |
| 102 | // a broker returns a response message larger than this value, Sarama will return a PacketDecodingError to |
| 103 | // protect the client from running out of memory. Please note that brokers do not have any natural limit on |
| 104 | // the size of responses they send. In particular, they can send arbitrarily large fetch responses to consumers |
| 105 | // (see https://issues.apache.org/jira/browse/KAFKA-2063). |
| 106 | MaxResponseSize int32 = 100 * 1024 * 1024 |
| 107 | ) |
| 108 | |
| 109 | // StdLogger is used to log error messages. |
| 110 | type StdLogger interface { |
| 111 | Print(v ...interface{}) |
| 112 | Printf(format string, v ...interface{}) |
| 113 | Println(v ...interface{}) |
| 114 | } |
kesavand | c71914f | 2022-03-25 11:19:03 +0530 | [diff] [blame] | 115 | |
| 116 | type debugLogger struct{} |
| 117 | |
| 118 | func (d *debugLogger) Print(v ...interface{}) { |
| 119 | Logger.Print(v...) |
| 120 | } |
| 121 | func (d *debugLogger) Printf(format string, v ...interface{}) { |
| 122 | Logger.Printf(format, v...) |
| 123 | } |
| 124 | func (d *debugLogger) Println(v ...interface{}) { |
| 125 | Logger.Println(v...) |
| 126 | } |
| 127 | |
| 128 | // DebugLogger is the instance of a StdLogger that Sarama writes more verbose |
| 129 | // debug information to. By default it is set to redirect all debug to the |
| 130 | // default Logger above, but you can optionally set it to another StdLogger |
| 131 | // instance to (e.g.,) discard debug information |
| 132 | var DebugLogger StdLogger = &debugLogger{} |