Etcd - 分布式配置中心

Posted on   |   In ,

Etcd 简介

Etcd 是一种分布式 kv 存储设施, 他具有一定的一致性,高性能,高可用的方案. 类似的 zookeeper, 但没有 zookeeper 那么重型,功能也没有覆盖那么多. 简单直接的应用就是配置中心

架构设计总览

clients 为多个需要  配置的服务, 中间层为 多个 grpc-proxy 做均衡负载, 以免一个 proxy 挂了之后  导致单点问题. grpc-proxy 可以  同时访问多个 etcd 服务器,进行 kv 的操作. 如果某一个 server  挂了,会自动访问别的  集群中的其他 server 以保证高可用.

etcd cluster 至少为 3 台, 如果小于 3 台则无法进行选举,造成集群  不可用. (这里需要用 promethus 进行监控预警)

etcd cluster 本地搭建

  1. 本地安装 etcd,或者直接下载预编译好的执行文件
    https://github.com/etcd-io/etcd/releases
  2. 获取 etcd 源码 (其实只是要一个配置文件 Procfile)
  3. 安装 goreman -> go get github.com/mattn/goreman
  4. 修改源码根目录的 Procfile
# Use goreman to run `go get github.com/mattn/goreman`
etcd1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr
etcd2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr
etcd3: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr
etcd4: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra4 --listen-client-urls http://127.0.0.1:42379 --advertise-client-urls http://127.0.0.1:42379 --listen-peer-urls http://127.0.0.1:42380 --initial-advertise-peer-urls http://127.0.0.1:42380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr
etcd5: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd --name infra5 --listen-client-urls http://127.0.0.1:52379 --advertise-client-urls http://127.0.0.1:52379 --listen-peer-urls http://127.0.0.1:52380 --initial-advertise-peer-urls http://127.0.0.1:52380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380,infra4=http://127.0.0.1:42380,infra5=http://127.0.0.1:52380' --initial-cluster-state new --enable-pprof --log-output=stderr
proxy1: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23790 --advertise-client-url=127.0.0.1:23790 --enable-pprof
proxy2: /Users/vincent/Downloads/etcd-v3.3.9-darwin-amd64/etcd grpc-proxy start --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 --listen-addr=127.0.0.1:23791 --advertise-client-url=127.0.0.1:23791 --enable-pprof
  1. 启动集群 goreman -f Procfile start

配置解读

配置了 5 个 etcd 服务模拟不同的 server, 配置了 2 个 proxy, 模拟两个 grpc-proxy 当做均衡负载
–listen-client-urls etcd 监听的地址
–listen-peer-urls 集群内部 member 互相监听的地址
–name 集群成员的名字
–advertise-client-urls 集群内成员对外发布信息的地址
–initial-advertise-peer-urls 对集群内发布信息的地址
–initial-cluster-state Set to new for all members present during initial static or DNS bootstrapping. If this option is set to existing, etcd will attempt to join the existing cluster. If the wrong value is set, etcd will attempt to start but fail safely.
–initial-cluster-token 集群的标示

5 个 etcd server 形成 etcd-cluster-1, 2 个 proxy 对这个集群进行代理

集群检查

ETCDCTL_API=3 ./etcdctl endpoint –endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table

λ ETCDCTL_API=3 ./etcdctl endpoint --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379 status -w table
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|    ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|  127.0.0.1:2379 | 8211f1d0f64f3269 |   3.3.9 |   25 kB |     false |         2 |         12 |
| 127.0.0.1:22379 | 91bc3c398fb3c146 |   3.3.9 |   25 kB |     false |         2 |         12 |
| 127.0.0.1:32379 | fd422379fda50e48 |   3.3.9 |   25 kB |     false |         2 |         12 |
| 127.0.0.1:42379 | 45d559f8148de837 |   3.3.9 |   25 kB |     false |         2 |         12 |
| 127.0.0.1:52379 | c91263fe1e1dd3b5 |   3.3.9 |   25 kB |      true |         2 |         12 |
+-----------------+------------------+---------+---------+-----------+-----------+------------+
λ ./etcdctl cluster-health
member 45d559f8148de837 is healthy: got healthy result from http://127.0.0.1:42379
member 8211f1d0f64f3269 is healthy: got healthy result from http://127.0.0.1:2379
member 91bc3c398fb3c146 is healthy: got healthy result from http://127.0.0.1:22379
member c91263fe1e1dd3b5 is healthy: got healthy result from http://127.0.0.1:52379
member fd422379fda50e48 is healthy: got healthy result from http://127.0.0.1:32379
cluster is healthy

编写测试客户端进行 kv 操作

go get go.etcd.io/etcd

package main

import (
	"context"
	"fmt"
	"log"
	"strconv"
	"time"

	"go.etcd.io/etcd/clientv3"
	"vincent.com/etcd/etcd/etcdserver/api/v3rpc/rpctypes"
)

func main() {

	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"http://127.0.0.1:23790", "http://127.0.0.1:23791"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		log.Fatalln(err.Error())
	}
	defer cli.Close()
	defer cli.Delete(context.Background(), "sample_key")

	for index := 0; index < 10000; index++ {
		insertKV(cli, "sample_key", "sample_value"+strconv.Itoa(index))
		if err != nil {
			break
		}
	}
	resp, err := cli.Get(context.Background(), "sample_key")
	if err != nil {
		log.Fatalln("get key error", err)
	}
	fmt.Printf("get the sample_key: %v\n", resp.Kvs)
}

func insertKV(cli *clientv3.Client, key string, value string) (err error) {
	time.Sleep(2 * time.Millisecond)
	_, err = cli.Put(context.Background(), key, value)
	// cancel()
	if err != nil {
		switch err {
		case context.Canceled:
			log.Fatalf("ctx is canceled by another routine: %v", err)

		case context.DeadlineExceeded:
			log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
		case rpctypes.ErrEmptyKey:
			log.Fatalf("client-side error: %v", err)
		default:
			log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
		}
	}
	return
}

客户端解读

etcd 客户端支持集群, 所以直接可以连接两个 proxy
这里进行了 1w 此的写入,每次写入会有 x ms 的延迟(我怕我的 mbp 受不了)

 客户端测试

  • 运行客户端程序
  • 模拟 etcd server  挂掉 goreman run stop etcd2
  • 当集群内健康机器少于 3 台的时候,客户端报错,集群整体不可用

集群性能检测

λ ETCDCTL_API=3 ./etcdctl check perf load="l" --endpoints=127.0.0.1:2379,127.0.0.1:22379,127.0.0.1:32379,127.0.0.1:42379,127.0.0.1:52379
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.388842s
PASS: Stddev is 0.029037s
PASS

注意:

如果用命令行,记得加 ETCDCTL_API=3