v3.3.0

See code changes and v3.3 upgrade guide for any breaking changes.

Improved

• Use coreos/bbolt to replace boltdb/bolt.
  • Fix etcd database size grows until mvcc: database space exceeded.
• Reduce memory allocation on Range operations.
• Rate limit and randomize lease revoke on restart or leader elections.
  • Prevent spikes in Raft proposal rate.
• Support clientv3 balancer failover under network faults/partitions.
• Better warning on mismatched --initial-cluster flag.

Changed(Breaking Changes)

• Require Go 1.9+.
  • Compile with Go 1.9.2.
  • Deprecate golang.org/x/net/context.
• Require google.golang.org/grpc v1.7.4 or v1.7.5+:
  • Deprecate metadata.Incoming/OutgoingContext.
  • Deprecate grpclog.Logger, upgrade to grpclog.LoggerV2.
  • Deprecate grpc.ErrClientConnTimeout errors in clientv3.
  • Use MaxRecvMsgSize and MaxSendMsgSize to limit message size, in etcd server.
• Upgrade github.com/grpc-ecosystem/grpc-gateway v1.2.2 to v1.3.0.
• Translate gRPC status error in v3 client Snapshot API.
• Upgrade github.com/ugorji/go/codec for v2 client.
  • Regenerated v2 client source code with latest ugorji/go/codec.
• Fix /health endpoint JSON output.
• v3 etcdctl lease timetolive LEASE_ID on expired lease now prints lease LEASE_ID already expired.
  • <=3.2 prints lease LEASE_ID granted with TTL(0s), remaining(-1s).

Added(etcd)

• Add --experimental-enable-v2v3 flag to emulate v2 API with v3.
• Add --experimental-corrupt-check-time flag to raise corrupt alarm monitoring.
• Add --experimental-initial-corrupt-check flag to check database hash before serving client/peer traffic.
• Add --max-txn-ops flag to configure maximum number operations in transaction.
• Add --max-request-bytes flag to configure maximum client request size.
  • If not configured, it defaults to 1.5 MiB.
• Add --client-crl-file, --peer-crl-file flags for Certificate revocation list.
• Add --peer-require-cn flag to support CN-based auth for inter-peer connection.
• Add --listen-metrics-urls flag for additional /metrics endpoints.
  • Support additional (non) TLS /metrics endpoints for a TLS-enabled cluster.
  • e.g. --listen-metrics-urls=https://localhost:2378,http://localhost:9379 to serve /metrics in secure port 2378 and insecure port 9379.
  • Useful for bypassing critical APIs when monitoring etcd.
• Add --auto-compaction-mode flag to support revision-based compaction.
• Change --auto-compaction-retention flag to accept string values with finer granularity.
• Add --grpc-keepalive-min-time, --grpc-keepalive-interval, --grpc-keepalive-timeout flags to configure server-side keepalive policies.
• Serve /health endpoint as unhealthy when alarm is raised.
• Provide error information in /health.
  • e.g. {"health":false,"errors":["NOSPACE"]}.
• Move logging setup to embed package
  • Disable gRPC server log by default.
• Use monotonic time in Go 1.9 for lease package.
• Warn on empty hosts in advertise URLs.
  • Address advertise client URLs accepts empty hosts.
  • etcd v3.4 will exit on this error.
    • e.g. --advertise-client-urls=http://:2379.
• Warn on shadowed environment variables.
  • Address error on shadowed environment variables.
  • etcd v3.4 will exit on this error.

Added(API)

• Support ranges in transaction comparisons for disconnected linearized reads.
• Add nested transactions to extend proxy use cases.
• Add lease comparison target in transaction.
• Add lease list.
• Add hash by revision for better corruption checking against boltdb.

Added(etcd/clientv3)

• Add health balancer to fix watch API hangs, improve endpoint switch under network faults.
• Refactor balancer and add client-side keepalive pings to handle network partitions.
• Add MaxCallSendMsgSize and MaxCallRecvMsgSize fields to clientv3.Config.
  • Fix exceeded response size limit error in client-side.
  • Address kubernetes#51099.
  • MaxCallSendMsgSize default value is 2 MiB, if not configured.
  • MaxCallRecvMsgSize default value is math.MaxInt32, if not configured.
• Accept Compare_LEASE in clientv3.Compare.
• Add LeaseValue helper to Cmp LeaseID values in Txn.
• Add MoveLeader to Maintenance.
• Add HashKV to Maintenance.
• Add Leases to Lease.
• Add clientv3/ordering for enforce ordering in serialized requests.

Added(v2 etcdctl)

• Add backup --with-v3 flag.

Added(v3 etcdctl)

• Add --discovery-srv flag.
• Add --keepalive-time, --keepalive-timeout flags.
• Add lease list command.
• Add lease keep-alive --once flag.
• Make lease timetolive LEASE_ID on expired lease print lease LEASE_ID already expired.
  • <=3.2 prints lease LEASE_ID granted with TTL(0s), remaining(-1s).
• Add defrag --data-dir flag.
• Add move-leader command.
• Add endpoint hashkv command.
• Add endpoint --cluster flag, equivalent to v2 etcdctl cluster-health.
• Make endpoint health command terminate with non-zero exit code on unhealthy status.
• Add lock --ttl flag.
• Support watch [key] [range_end] -- [exec-command…], equivalent to v2 etcdctl exec-watch.
• Enable clientv3.WithRequireLeader(context.Context) for watch command.
• Print "del" instead of "delete" in txn interactive mode.
• Print ETCD_INITIAL_ADVERTISE_PEER_URLS in member add.

Added(metrics)

• Add etcd --listen-metrics-urls flag for additional /metrics endpoints.
  • Useful for bypassing critical APIs when monitoring etcd.
• Add etcd_server_version Prometheus metric.
  • To replace Kubernetes etcd-version-monitor.
• Add etcd_debugging_mvcc_db_compaction_keys_total Prometheus metric.
• Add etcd_debugging_server_lease_expired_total Prometheus metric.
  • To improve lease revoke monitoring.
• Document Prometheus 2.0 rules.
• Initialize gRPC server metrics with zero values.

Added(grpc-proxy)

• Add grpc-proxy start --experimental-leasing-prefix flag:
  • For disconnected linearized reads.
  • Based on V system leasing.
  • See "Disconnected consistent reads with etcd" blog post.
• Add grpc-proxy start --experimental-serializable-ordering flag.
  • To ensure serializable reads have monotonically increasing store revisions across endpoints.
• Add grpc-proxy start --metrics-addr flag for an additional /metrics endpoint.
  • Set --metrics-addr=http://[HOST]:9379 to serve /metrics in insecure port 9379.
• Serve /health endpoint in grpc-proxy.
• Add grpc-proxy start --debug flag.

Added(gRPC gateway)

• Replace gRPC gateway endpoint with /v3beta.
  • To deprecate /v3alpha in v3.4.
• Support "authorization" token.
• Support websocket for bi-directional streams.
  • Fix Watch API with gRPC gateway.
• Upgrade gRPC gateway to v1.3.0.

Added(etcd/raft)

• Add non-voting member.
  • To implement Raft thesis 4.2.1 Catching up new servers.
  • Learner node does not vote or promote itself.

Added/Fixed(Security/Auth)

• Add CRL based connection rejection to manage revoked certs.
• Document TLS authentication changes:
  • Server accepts connections if IP matches, without checking DNS entries. For instance, if peer cert contains IP addresses and DNS names in Subject Alternative Name (SAN) field, and the remote IP address matches one of those IP addresses, server just accepts connection without further checking the DNS names.
  • Server supports reverse-lookup on wildcard DNS SAN. For instance, if peer cert contains only DNS names (no IP addresses) in Subject Alternative Name (SAN) field, server first reverse-lookups the remote IP address to get a list of names mapping to that address (e.g. nslookup IPADDR). Then accepts the connection if those names have a matching name with peer cert's DNS names (either by exact or wildcard match). If none is matched, server forward-lookups each DNS entry in peer cert (e.g. look up example.default.svc when the entry is *.example.default.svc), and accepts connection only when the host's resolved addresses have the matching IP address with the peer's remote IP address.
• Add etcd --peer-require-cn flag.
  • To support CommonName(CN) based auth for inter peer connection.
• Swap priority of cert CommonName(CN) and username + password.
  • To address "username and password specified in the request should take priority over CN in the cert".
• Protect lease revoke with auth.
• Provide user's role on auth permission error.
• Fix auth store panic with disabled token.
• Update golang.org/x/crypto/bcrypt (see golang/crypto@6c586e1).

Fixed(v2)

• Fail-over v2 client to next endpoint on oneshot failure.
• Put back /v2/machines endpoint for python-etcd wrapper.

Fixed(v3)

• Fix range/put/delete operation metrics with transaction:
  • etcd_debugging_mvcc_range_total
  • etcd_debugging_mvcc_put_total
  • etcd_debugging_mvcc_delete_total
  • etcd_debugging_mvcc_txn_total
• Fix etcd_debugging_mvcc_keys_total on restore.
• Fix etcd_debugging_mvcc_db_total_size_in_bytes on restore.
  • Also change to prometheus.NewGaugeFunc.
• Fix backend database in-memory index corruption issue on restore (only 3.2.0 is affected).
• Fix watch restore from snapshot.
• Fix "put at-most-once" in clientv3.
• Handle empty key permission in etcdctl.
• Fix server crash on invalid transaction request from gRPC gateway.
• Fix clientv3.WatchResponse.Canceled on compacted watch request.
• Handle WAL renaming failure on Windows.
• Make peer dial timeout longer.
  • See coreos/etcd-operator#1300 for more detail.
• Make server wait up to request time-out with pending RPCs.
• Fix grpc.Server panic on GracefulStop with TLS-enabled server.
• Fix "multiple peer URLs cannot start" issue.
• Fix server-side auth so concurrent auth operations do not return old revision error.
• Fix concurrency/stm Put with serializable snapshot.
  • Use store revision from first fetch to resolve write conflicts instead of modified revision.
• Fix grpc-proxy Snapshot API error handling.
• Fix grpc-proxy KV API PrevKv flag handling.
• Fix grpc-proxy KV API KeysOnly flag handling.
• Upgrade coreos/go-systemd to v15 (see https://github.com/coreos/go-systemd/releases/tag/v15).

Other

• Support previous two minor versions (see our new release policy).
• v3.3.x is the last release cycle that supports ACI:
  • AppC was officially suspended, as of late 2016.
  • acbuild is not maintained anymore.
  • *.aci files won't be available from etcd v3.4 release.
• Add container registry gcr.io/etcd-development/etcd.
  • quay.io/coreos/etcd is still supported as secondary.

v3.2.12 (2017-12-20)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Fix error message of Revision compactor in server-side.

Added(etcd/clientv3,etcdctl/v3)

• Add MaxCallSendMsgSize and MaxCallRecvMsgSize fields to clientv3.Config.
  • Fix exceeded response size limit error in client-side.
  • Address kubernetes#51099.
  • MaxCallSendMsgSize default value is 2 MiB, if not configured.
  • MaxCallRecvMsgSize default value is math.MaxInt32, if not configured.

Other

• Pin grpc v1.7.5, grpc-gateway v1.3.0.
  • No code change, just to be explicit about recommended versions.

v3.2.11 (2017-12-05)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Fix racey grpc-go's server handler transport WriteStatus call to prevent TLS-enabled etcd server crash:
  • Upgrade google.golang.org/grpc v1.7.3 to v1.7.4.
  • Add gRPC RPC failure warnings to help debug such issues in the future.
• Remove --listen-metrics-urls flag in monitoring document (non-released in v3.2.x, planned for v3.3.x).

Added

• Provide more cert details on TLS handshake failures.

v3.1.11 (2017-11-28)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• #8411,#8806 mvcc: fix watch restore from snapshot
• #8009,#8902 backport coreos/bbolt v1.3.1-coreos.5

v3.2.10 (2017-11-16)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Replace backend key-value database boltdb/bolt with coreos/bbolt to address backend database size issue.
• Fix clientv3 balancer to handle network partitions:
  • Upgrade google.golang.org/grpc v1.2.1 to v1.7.3.
  • Upgrade github.com/grpc-ecosystem/grpc-gateway v1.2 to v1.3.
• Revert discovery SRV auth ServerName with *.{ROOT_DOMAIN} to support non-wildcard subject alternative names in the certs (see issue #8445 for more contexts).
  • For instance, etcd --discovery-srv=etcd.local will only authenticate peers/clients when the provided certs have root domain etcd.local (not *.etcd.local) as an entry in Subject Alternative Name (SAN) field.

v3.2.9 (2017-10-06)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed(Security)

• Compile with Go 1.8.4.
• Update golang.org/x/crypto/bcrypt (see golang/crypto@6c586e1).
• Fix discovery SRV bootstrapping to authenticate ServerName with *.{ROOT_DOMAIN}, in order to support sub-domain wildcard matching (see issue #8445 for more contexts).
  • For instance, etcd --discovery-srv=etcd.local will only authenticate peers/clients when the provided certs have root domain *.etcd.local as an entry in Subject Alternative Name (SAN) field.

v3.2.8 (2017-09-29)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Fix v2 client failover to next endpoint on mutable operation.
• Fix grpc-proxy to respect KeysOnly flag.

v3.2.7 (2017-09-01)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Fix server-side auth so concurrent auth operations do not return old revision error.
• Fix concurrency/stm Put with serializable snapshot
  • Use store revision from first fetch to resolve write conflicts instead of modified revision.

v3.2.6 (2017-08-21)

See code changes.

Fixed

• Fix watch restore from snapshot.
• Fix etcd_debugging_mvcc_keys_total inconsistency.
• Fix multiple URLs for --listen-peer-urls flag.
• Add --enable-pprof flag to etcd configuration file format.

v3.2.5 (2017-08-04)

See code changes and v3.2 upgrade guide for any breaking changes.

Changed

• Use reverse lookup to match wildcard DNS SAN.
• Return non-zero exit code on unhealthy endpoint health.

Fixed

• Fix unreachable /metrics endpoint when --enable-v2=false.
• Fix grpc-proxy to respect PrevKv flag.

Added

• Add container registry gcr.io/etcd-development/etcd.

v3.2.4 (2017-07-19)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Do not block on active client stream when stopping server
• Fix gRPC proxy Snapshot RPC error handling

v3.2.3 (2017-07-14)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Let clients establish unlimited streams

Added

• Tag docker images with minor versions
  • e.g. docker pull quay.io/coreos/etcd:v3.2 to fetch latest v3.2 versions

v3.1.10 (2017-07-14)

See code changes and v3.1 upgrade guide for any breaking changes.

Changed

• Compile with Go 1.8.3 to fix panic on net/http.CloseNotify

Added

• Tag docker images with minor versions.
  • e.g. docker pull quay.io/coreos/etcd:v3.1 to fetch latest v3.1 versions.

v3.2.2 (2017-07-07)

See code changes and v3.2 upgrade guide for any breaking changes.

Improved

• Rate-limit lease revoke on expiration.
• Extend leases on promote to avoid queueing effect on lease expiration.

Fixed

• Use user-provided listen address to connect to gRPC gateway:
  • net.Listener rewrites IPv4 0.0.0.0 to IPv6 [::], breaking IPv6 disabled hosts.
  • Only v3.2.0, v3.2.1 are affected.
• Accept connection with matched IP SAN but no DNS match.
  • Don't check DNS entries in certs if there's a matching IP.
• Fix 'tools/benchmark' watch command.

v3.2.1 (2017-06-23)

See code changes and v3.2 upgrade guide for any breaking changes.

Fixed

• Fix backend database in-memory index corruption issue on restore (only 3.2.0 is affected).
• Fix gRPC gateway Txn marshaling issue.
• Fix backend database size debugging metrics.

v3.2.0 (2017-06-09)

See code changes and v3.2 upgrade guide for any breaking changes.

Improved

• Improve backend read concurrency.

Added

• Embedded etcd
  • Etcd.Peers field is now []*peerListener.
• RPCs
  • Add Election, Lock service.
• Native client etcdserver/api/v3client
  • client "embedded" in the server.
• gRPC proxy
  • Proxy endpoint discovery.
  • Namespaces.
  • Coalesce lease requests.
• v3 client
  • STM prefetching.
  • Add namespace feature.
  • Add ErrOldCluster with server version checking.
  • Translate WithPrefix() into WithFromKey() for empty key.
• v3 etcdctl
  • Add check perf command.
  • Add --from-key flag to role grant-permission command.
  • lock command takes an optional command to execute.
• etcd flags
  • Add --enable-v2 flag to configure v2 backend (enabled by default).
  • Add --auth-token flag.
• etcd gateway
  • Support DNS SRV priority.
• Auth
  • Support Watch API.
  • JWT tokens.
• Logging, monitoring
  • Server warns large snapshot operations.
  • Add etcd_debugging_server_lease_expired_total metrics.
• Security
  • Deny incoming peer certs with wrong IP SAN.
  • Resolve TLS DNSNames when SAN checking.
  • Reload TLS certificates on every client connection.
• Release
  • Annotate acbuild with supports-systemd-notify.
  • Add nsswitch.conf to Docker container image.
  • Add ppc64le, arm64(experimental) builds.
  • Compile with Go 1.8.3.

Changed

• v3 client
  • LeaseTimeToLive returns TTL=-1 resp on lease not found.
  • clientv3.NewFromConfigFile is moved to clientv3/yaml.NewConfig.
  • concurrency package's elections updated to match RPC interfaces.
  • let client dial endpoints not in the balancer.
• Dependencies
  • Update google.golang.org/grpc to v1.2.1.
  • Update github.com/grpc-ecosystem/grpc-gateway to v1.2.0.

Fixed

• Allow v2 snapshot over 512MB.

v3.1.9 (2017-06-09)

See code changes and v3.1 upgrade guide for any breaking changes.

Fixed

• Allow v2 snapshot over 512MB.

v3.1.8 (2017-05-19)

See code changes and v3.1 upgrade guide for any breaking changes.

v3.1.7 (2017-04-28)

See code changes and v3.1 upgrade guide for any breaking changes.

v3.1.6 (2017-04-19)

See code changes and v3.1 upgrade guide for any breaking changes.

Changed

• Remove auth check in Status API.

Fixed

• Fill in Auth API response header.

v3.1.5 (2017-03-27)

See code changes and v3.1 upgrade guide for any breaking changes.

Added

• Add /etc/nsswitch.conf file to alpine-based Docker image.

Fixed

• Fix raft memory leak issue.
• Fix Windows file path issues.

v3.1.4 (2017-03-22)

See code changes and v3.1 upgrade guide for any breaking changes.

v3.1.3 (2017-03-10)

See code changes and v3.1 upgrade guide for any breaking changes.

Changed

• Use machine default host when advertise URLs are default values(localhost:2379,2380) AND if listen URL is 0.0.0.0.

Fixed

• Fix etcd gateway schema handling in DNS discovery.
• Fix sd_notify behaviors in gateway, grpc-proxy.

v3.1.2 (2017-02-24)

See code changes and v3.1 upgrade guide for any breaking changes.

Changed

• Use IPv4 default host, by default (when IPv4 and IPv6 are available).

Fixed

• Fix etcd gateway with multiple endpoints.

v3.1.1 (2017-02-17)

See code changes and v3.1 upgrade guide for any breaking changes.

Changed

• Compile with Go 1.7.5.

v2.3.8 (2017-02-17)

See code changes.

Changed

• Compile with Go 1.7.5.

v3.1.0 (2017-01-20)

See code changes and v3.1 upgrade guide for any breaking changes.

Improved

• Faster linearizable reads (implements Raft read-index).
• v3 authentication API is now stable.

Added

• Automatic leadership transfer when leader steps down.
• etcd flags
  • --strict-reconfig-check flag is set by default.
  • Add --log-output flag.
  • Add --metrics flag.
• v3 client
  • Add SetEndpoints method; update endpoints at runtime.
  • Add Sync method; auto-update endpoints at runtime.
  • Add Lease TimeToLive API; fetch lease information.
  • replace Config.Logger field with global logger.
  • Get API responses are sorted in ascending order by default.
• v3 etcdctl
  • Add lease timetolive command.
  • Add --print-value-only flag to get command.
  • Add --dest-prefix flag to make-mirror command.
  • get command responses are sorted in ascending order by default.
• recipes now conform to sessions defined in clientv3/concurrency.
• ACI has symlinks to /usr/local/bin/etcd*.
• Experimental gRPC proxy feature.

Changed

• Deprecated following gRPC metrics in favor of go-grpc-prometheus:
  • etcd_grpc_requests_total
  • etcd_grpc_requests_failed_total
  • etcd_grpc_active_streams
  • etcd_grpc_unary_requests_duration_seconds
• etcd uses default route IP if advertise URL is not given.
• Cluster rejects removing members if quorum will be lost.
• SRV records (e.g., infra1.example.com) must match the discovery domain (i.e., example.com) if no custom certificate authority is given.
  • TLSConfig.ServerName is ignored with user-provided certificates for backwards compatibility; to be deprecated.
  • For example, etcd --discovery-srv=example.com will only authenticate peers/clients when the provided certs have root domain example.com as an entry in Subject Alternative Name (SAN) field.
• Discovery now has upper limit for waiting on retries.
• Warn on binding listeners through domain names; to be deprecated.

v3.0.16 (2016-11-13)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.15 (2016-11-11)

See code changes and v3.0 upgrade guide for any breaking changes.

Fixed

• Fix cancel watch request with wrong range end.

v3.0.14 (2016-11-04)

See code changes and v3.0 upgrade guide for any breaking changes.

Added

• v3 etcdctl migrate command now supports --no-ttl flag to discard keys on transform.

v3.0.13 (2016-10-24)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.12 (2016-10-07)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.11 (2016-10-07)

See code changes and v3.0 upgrade guide for any breaking changes.

Added

• Server returns previous key-value (optional)
  • clientv3.WithPrevKV option
  • v3 etcdctl put,watch,del --prev-kv flag

v3.0.10 (2016-09-23)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.9 (2016-09-15)

See code changes and v3.0 upgrade guide for any breaking changes.

Added

• Warn on domain names on listen URLs (v3.2 will reject domain names).

v3.0.8 (2016-09-09)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• Allow only IP addresses in listen URLs (domain names are rejected).

v3.0.7 (2016-08-31)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• SRV records only allow A records (RFC 2052).

v3.0.6 (2016-08-19)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.5 (2016-08-19)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• SRV records (e.g., infra1.example.com) must match the discovery domain (i.e., example.com) if no custom certificate authority is given.

v3.0.4 (2016-07-27)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• v2 auth can now use common name from TLS certificate when --client-cert-auth is enabled.

Added

• v2 etcdctl ls command now supports --output=json.
• Add /var/lib/etcd directory to etcd official Docker image.

v3.0.3 (2016-07-15)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• Revert Dockerfile to use CMD, instead of ENTRYPOINT, to support etcdctl run.
  • Docker commands for v3.0.2 won't work without specifying executable binary paths.
• v3 etcdctl default endpoints are now 127.0.0.1:2379.

v3.0.2 (2016-07-08)

See code changes and v3.0 upgrade guide for any breaking changes.

Changed

• Dockerfile uses ENTRYPOINT, instead of CMD, to run etcd without binary path specified.

v3.0.1 (2016-07-01)

See code changes and v3.0 upgrade guide for any breaking changes.

v3.0.0 (2016-06-30)

See code changes and v3.0 upgrade guide for any breaking changes.