基于隧道的Kubernetes跨集群通讯

CNCF

发布于 2023-02-12 17:13:13

1.8K0

发布于 2023-02-12 17:13:13

文章被收录于专栏：CNCF

伴随着k8s的大量使用，无论是基于应用隔离或者高可用，容灾的需要还是运维管理的需求，很多企业都会部署多个K8S集群。这就会导致有些应用依赖于其它k8s集群的微服务，需要从一个集群里的pod访问另外一个集群里的pod或者service。为了解决跨集群服务调用的问题，我们试验了一种基于隧道的方案，下面就让我们一起来体验一下吧。

作者：鲍盈海，中国移动云能力中心软件开发工程师，专注于云原生领域。

环境要求：

组件	版本	备注
kubernetes	1.21.5以上(包含)	要求有两个集群，且至少其中一个集群的node节点可以访问另外一个集群中的某一个node节点。
docker	18.09.5	打镜像
go	1.19	用于开发隧道代理及模拟业务服务

单隧道单服务访问

先介绍一下最简单的场景，让集群A中的服务访问集群B中的服务，架构图如下：

集群A和集群B由一条隧道连接，隧道的左边是代理了隧道入口的service，隧道的右边是一个业务服务，我们在左边集群A中在宿主机上通过curl+ip的方式（或者在容器中通过域名的方式）访问集群B的业务服务。下面我们一起来实际操作一下。我们从右往左来部署服务。

1.先部署demo-service，这个服务是golang官网的demo : https://go.dev/doc/tutorial/web-service-gin, 将它打包成镜像后使用k8s来部署在集群B中充当业务服务，创建pod和svc的yaml文件如下 :

---
apiVersion: v1
kind: Namespacemetadata:
  name: tunnel-proxy
 
---
apiVersion: apps/v1
kind: Deploymentmetadata:
  name: demo-service
  namespace: tunnel-proxyspec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-service
  template:
    metadata:
      labels:
        app: demo-service
    spec:
      containers:
      - name: demo-service
        image: nexus.cmss.com:8086/cnp/tunnel/demo-service:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http-console
 
---
apiVersion: v1
kind: Servicemetadata:
  name: tunnel-proxy
  namespace: tunnel-proxyspec:
  type: NodePort
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 31080
  selector:
    app: demo-service
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack

2. 创建隧道，我们使用ssh命令来创建一个ssh隧道，在clusterB上执行如下的命令：

ssh -NR *:8079:localhost:31080 root@[集群A的隧道入口机器IP]

其中8079是集群A上监听的端口，31080是集群B上监听的端口，也是demo-service的svc暴露出来的端口，后面root@[集群A的隧道入口机器IP]是集群A中的机器。需要注意的是集群A的机器的ssh必须开启网关转发功能，具体修改步骤是将/etc/ssh/sshd_config文件中的GatewayPorts改为yes，并重启sshd。

3. 我们还创建了一个tunnel-service的svc，这是一个没有selector的服务，目的是方便集群A中的服务来访问隧道。详细的yaml如下：

---
apiVersion: v1
kind: Namespacemetadata:
  name: tunnel-proxy
 
---
apiVersion: v1
kind: Servicemetadata:
  name: tunnel-proxy
  namespace: tunnel-proxyspec:
  type: NodePort
  ports:
    - protocol: TCP
      port: 8079
      targetPort: 8079
      nodePort: 31079
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
 
apiVersion: v1
kind: Endpointsmetadata:
  name: tunnel-proxy
  namespace: tunnel-proxysubsets:
  - addresses:
      - ip: [集群A的隧道入口机器IP]
    ports:
      - port: 8079

至此，在集群A中就可以通过隧道来访问集群B的服务了。但是目前这种方案还不适合在生产环境中使用，因为一般情况下跨集群之间因为安全、性能、成本等因素多个服务会复用一条隧道，而上面的方案中，一个服务独占了一条隧道，如果有多个demo-service服务则需要创建多个隧道。所以我们设计了单隧道多服务的方案。

单隧道多服务访问

单隧道多服务的实现原理是在隧道两头增加一个隧道的代理，隧道左端监听多个端口，用来区分集群A中服务要访问的集群B中的不同服务。并且将此信息告知隧道游段的代理，隧道右段代理根据此信息来转发给对应的ClusterB中的服务。架构图如下：

我们定义了一个配置文件，来描述隧道左侧监听的端口与隧道右侧服务映射的关系，如下:

{
  "data": [
    {
      "port": "8050", 
      "remoteIP": "http://demo-service-1.cnp-tunnel.svc.cluster.local:8080"
    },
        {
      "port": "8051",
      "remoteIP": "http://demo-service-2.cnp-tunnel.svc.cluster.local:8080"
    }
  ]}

这个json文件中描述了两条映射关系，集群A隧道左端8050端口接收到的请求都会转给集群B的http://demo-service-1.cnp-tunnel.svc.cluster.local:8080即biz-1f服务，同理8051端口的请求转给biz-2。在实现过程中，当隧道左端接收到请求时，会在当前请求的header中新增一个名为“X-Proxy-Condition”的信息，记录了当前请求来自哪个端口，在隧道右端读取到这个信息后就知道要转发给集群B中的哪个服务了。

关键代码列举

上面配置的功能是被隧道发送方和接收方共享的，所以在代码设计中做了三个module，分别是common（读取配置文件），receive（隧道右侧代理），send（隧道左侧代理）。目录结构如下：

其中三个go文件和go.mod文件如下：

common/config.go

package common
 
import (
    "encoding/json"
    "log"
    "os"
)
// Port是send端启动的时候 创建的服务端口， 集群A通过端口区分访问的服务// Port对应的RemoteIP是集群B中服务的地址，
type Config struct {
    Port     string `json:"port"`
    RemoteIP string `json:"remoteIp"`
}
 
type ConfigHelper struct {
    Data []Config `json:"data"`
}
 
func (configHelper *ConfigHelper) loadJson() error {
    // 从文件读取
    jsonFile, err := os.Open("config.json")
    if err != nil {
        log.Fatalln("Cannot open config file", err)
    }
 
    defer jsonFile.Close()
 
    decoder := json.NewDecoder(jsonFile)
    err = decoder.Decode(&configHelper)
    if err != nil {
        log.Fatalln("Cannot get configuration from file", err)
        return err
    }
    return nil
}
// 从配置文件读取
func (configHelper *ConfigHelper) GetConfigIns() ([]Config, error) {
    if configHelper.Data == nil {
        err := configHelper.loadJson()
        if err != nil {
            return nil, err
        }
    }
    log.Println(configHelper)
    return configHelper.Data, nil
}

common/go.mod

module tunnel/http-proxy/common
 go 1.19

receive/main.go

package main
import (
    "errors"
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "strings"
    "tunnel/http-proxy/common"
)
const PORT = "8080"
var configHelper = &common.ConfigHelper{}
 
type requestPayloadStruct struct {
    ProxyCondition string `json:"proxy_condition"`
}
// Get the port to listen onfunc getListenAddress() string {
    return ":" + PORT
}
// Log the env variables required for a reverse proxyfunc logSetup() {
    log.Printf("Server will run on: %s\n", getListenAddress())
}
// Log the typeform payload and redirect urlfunc logRequestPayload(requestionPayload requestPayloadStruct, proxyUrl string) {
    log.Printf("proxy_condition: %s, proxy_url: %s\n", requestionPayload.ProxyCondition, proxyUrl)
}
// Get the url for a given proxy conditionfunc getProxyUrl(proxyConditionRaw string) (string, error) {
    proxyCondition := strings.ToUpper(proxyConditionRaw)
 
    configIns, err := configHelper.GetConfigIns()
    if err != nil {
        log.Fatalln("proxy config is nil", err)
        return "", errors.New("not match config")
    }
    for i := 0; i < len(configIns); i++ {
        if configIns[i].Port == proxyCondition {
            return configIns[i].RemoteIP, nil
        }
    }
    return "", errors.New("not match config")
}
// Serve a reverse proxy for a given urlfunc serveReverseProxy(target string, res http.ResponseWriter, req *http.Request) {
    url, _ := url.Parse(target)
 
    proxy := httputil.NewSingleHostReverseProxy(url)
 
    proxy.ServeHTTP(res, req)
}
// Given a request send it to the appropriate urlfunc handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
    requestPayload := requestPayloadStruct{
        ProxyCondition: req.Header.Get("X-Proxy-Condition"),
    }
 
    url, err := getProxyUrl(requestPayload.ProxyCondition)
    if err != nil {
        log.Fatalln("proxy url error:", err)
        return
    }
 
    logRequestPayload(requestPayload, url)
 
    serveReverseProxy(url, res, req)
}
func main() {
    // Log setup values
    logSetup()
 
    // start server
    http.HandleFunc("/", handleRequestAndRedirect)
    if err := http.ListenAndServe(getListenAddress(), nil); err != nil {
        panic(err)
    }
}

receive/go.mod

module tunnel/http-proxy/receive
go 1.19
replace tunnel/http-proxy/common => ../common
require tunnel/http-proxy/common v0.0.0-00010101000000-000000000000

send/main.go

package main
import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "tunnel/http-proxy/common"
)
const TUNNEL_ENTER = "http://tunnel-proxy.cnp-tunnel.svc.cluster.local:8079"
var configHelper = &common.ConfigHelper{}
 
type requestPayloadStruct struct {
    ProxyCondition string `json:"proxy_condition"`
}
// Get the port to listen onfunc getListenAddress(port string) string {
    return ":" + port
}
// Log the env variables required for a reverse proxyfunc logSetup(configIns common.Config) {
    log.Printf("Server will run on: %s\n", getListenAddress(configIns.Port))
}
// Log the typeform payload and redirect urlfunc logRequestPayload(condition string, proxyUrl string) {
    log.Printf("proxy_condition: %s, proxy_url: %s\n", condition, proxyUrl)
}
// Serve a reverse proxy for a given urlfunc serveReverseProxy(target string, res http.ResponseWriter, req *http.Request) {
    // parse the url
    url, _ := url.Parse(target)
 
    // create the reverse proxy
    proxy := httputil.NewSingleHostReverseProxy(url)
 
    // Note that ServeHttp is non blocking and uses a go routine under the hood
    proxy.ServeHTTP(res, req)
}
// Given a request send it to the appropriate urlfunc handleRequestAndRedirect(res http.ResponseWriter, req *http.Request, condition string) {
    req.Header.Add("X-Proxy-Condition", condition)
 
    url := TUNNEL_ENTER
 
    logRequestPayload(condition, url)
 
    serveReverseProxy(url, res, req)
}
 
func main() {
    configIns, err := configHelper.GetConfigIns()
 
    if err != nil {
        log.Fatalln("proxy config is nil", err)
        return
    }
 
    // var httpMux *http.ServeMux
    for i := 0; i < len(configIns); i++ {
        // Log setup values
        logSetup(configIns[i])
        // start server
        port := configIns[i].Port
 
        httpMux := http.NewServeMux()
        httpMux.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) {
            handleRequestAndRedirect(res, req, port)
        })
        server := &http.Server{
            Addr:    getListenAddress(port),
            Handler: httpMux,
        }
        go server.ListenAndServe()
    }
 
    select {}
}

send/go.mod

module tunnel/http-proxy/send
go 1.19
replace tunnel/http-proxy/common => ../common
require tunnel/http-proxy/common v0.0.0-00010101000000-000000000000

构建tunnel-service和tunnel-sned的dockerfile文件分别如下：

FROM golang:1.19.5-alpine

WORKDIR /opt

ADD . /opt# 设置代理
RUN go env -w GO111MODULE=on
RUN go env -w GOPROXY=https://goproxy.io,direct

WORKDIR /opt/receive
 
RUN go build -o main ./main.go
 
EXPOSE 8080
 
CMD ["/opt/receive/main"]

FROM golang:1.19.5-alpine
 
WORKDIR /opt
 
ADD . /opt# 设置代理
RUN go env -w GO111MODULE=on
RUN go env -w GOPROXY=https://goproxy.io,direct
 
WORKDIR /opt/send
 
RUN go build -o main ./main.go
 
EXPOSE 8080
 
CMD ["/opt/send/main"]

打包镜像的脚本可以参考如下：

#!/bin/bashecho "building send\n"
 
docker build -f Dockerfile.send -t nexus.cmss.com:8086/cnp/tunnel/send:v1.0.0 .
if [ $? -ne 0 ]; then
  echo "build send failed\n"
  exit 1fi
 
docker push nexus.cmss.com:8086/cnp/tunnel/send:v1.0.0if [ $? -ne 0 ]; then
  echo "push send failed\n"
  exit 1fi
echo "build send success\n"
echo "building receive\n"
 
docker build -f Dockerfile.receive -t nexus.cmss.com:8086/cnp/tunnel/receive:v1.0.0 .
if [ $? -ne 0 ]; then
  echo "build receive failed\n"
  exit 1fi
 
docker push nexus.cmss.com:8086/cnp/tunnel/receive:v1.0.0if [ $? -ne 0 ]; then
  echo "push receive failed\n"
  exit 1fi
echo "build receive success\n"

最终我们打包了nexus.cmss.com:8086/cnp/tunnel/receive:v1.0.0和nexus.cmss.com:8086/cnp/tunnel/send:v1.0.0两个镜像。

部署实操

下面我们也来一起部署一下，同样是从右往左部署：

1.我们依然使用golang官网的demo : https://go.dev/doc/tutorial/web-service-gin, 作为demo-service，不同的是我们要创建两个。yaml文件如下：

---
apiVersion: v1
kind: Namespacemetadata:
  name: cnp-tunnel
# biz-1
---
apiVersion: apps/v1
kind: Deploymentmetadata:
  name: demo-service-1
  namespace: cnp-tunnelspec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-service-1
  template:
    metadata:
      labels:
        app: demo-service-1
    spec:
      containers:
      - name: demo-service-1
        image: nexus.cmss.com:8086/cnp/tunnel/demo-service:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http-console
 
---
apiVersion: v1
kind: Servicemetadata:
  name: demo-service-1
  namespace: cnp-tunnelspec:
  type: NodePort
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 31050
  selector:
    app: demo-service-1
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
# biz-2
---
apiVersion: apps/v1
kind: Deploymentmetadata:
  name: demo-service-2
  namespace: cnp-tunnelspec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-service-2
  template:
    metadata:
      labels:
        app: demo-service-2
    spec:
      containers:
      - name: demo-service-2
        image: nexus.cmss.com:8086/cnp/tunnel/demo-service:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http-console
 
---
apiVersion: v1
kind: Servicemetadata:
  name: demo-service-2
  namespace: cnp-tunnelspec:
  type: NodePort
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 31051
  selector:
    app: demo-service-2
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack

然后我们需要造一点数据，来区分两个服务，在这个demo中支持创建数据，命令如下：

# biz-1
curl 'http://[集群B中宿主机IP]:31050/albums' \
  -H 'content-type: application/json' \
  --data-raw '{"id":"4","title": "8050-add", "artist":"biz-1", "price": 100}' \
  --compressed \
  --insecure
# biz-2
 
curl 'http://[集群B中宿主机IP]:31051/albums' \
  H 'content-type: application/json' \
  --data-raw '{"id":"4","title": "8051-add", "artist":"biz-2", "price": 99}' \
  --compressed \
  --insecure

然后在浏览器中输入http://[集群B中宿主机IP]31050/albums

和http://[集群B中宿主机IP]:31051/albums来查看插入的数据是否生效。

2. 部署tunnel-receive服务，即隧道接收端的服务，其中namespace已经在第一步中创建了，yaml文件如下：

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tunnel-config
  namespace: cnp-tunnel
data:
  config.json: |
    {
      "data": [
        {
          "port": "8050", 
          "remoteIP": "http://localhost:31050"
        },
        {
          "port": "8051",
          "remoteIP": "http://localhost:31051"
        }
      ]
    }
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tunnel-receive
  namespace: cnp-tunnel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tunnel-receive
  template:
    metadata:
      labels:
        app: tunnel-receive
    spec:
      containers:
      - name: tunnel-receive
        image: nexus.cmss.com:8086/cnp/tunnel/receive:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: tunnel
        volumeMounts:
        - mountPath: /opt/send/config.json
          name: tunnel-config
          subPath: config.json
      volumes:
      - name: tunnel-config
        configMap:
          name: tunnel-config
 
---
apiVersion: v1
kind: Service
metadata:
  name: tunnel-receive
  namespace: cnp-tunnel
spec:
  type: NodePort
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
    nodePort: 31080
  selector:
    app: tunnel-receive
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack

服务部署后，可以通过在clusterB上执行下面的命令来检查receive服务是否正常，注意此处已经在header中设置了X-Proxy-Condition。

curl 'http://[集群B中宿主机IP]:31080/albums' \
  -H 'X-Proxy-Condition: 8050' \
  --compressed
 
curl 'http://[集群B中宿主机IP]:31080/albums' \
  -H 'X-Proxy-Condition: 8051' \
  --compressed

3. 创建隧道，同单隧道单服务中的步骤，执行下面的命令

ssh -NR *:8079:localhost:31080 root@[集群A的隧道入口机器IP]

在集群A上执行下面的命令来检查隧道是否成功创建：

curl 'http://[集群A的隧道入口机器IP]:8079/albums' \
  -H 'X-Proxy-Condition: 8050' \
  --compressed
 
curl 'http://[集群A的隧道入口机器IP]:8079/albums' \
  -H 'X-Proxy-Condition: 8051' \
  --compressed

同样的我们还需要创建一个没有selector的svc来代理隧道的左边，yaml文件如下：

---
apiVersion: v1
kind: Servicemetadata:
  name: tunnel-proxy
  namespace: cnp-tunnelspec:
  # type: NodePort
  ports:
    - protocol: TCP
      port: 8079
      targetPort: 8079
      # nodePort: 31079
  # externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
 
---
apiVersion: v1
kind: Endpointsmetadata:
  name: tunnel-proxy
  namespace: cnp-tunnelsubsets:
  - addresses:
      - ip: 100.76.11.99
    ports:
      - port: 8079

4. 部署tunnel-send服务，即隧道发送端的服务，yaml文件如下：

---
apiVersion: v1
kind: Namespace
metadata:
  name: cnp-tunnel
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tunnel-config
  namespace: cnp-tunnel
data:
  config.json: |
    {
      "data": [
        {
          "port": "8050", 
          "remoteIP": "http://localhost:31050"
        },
        {
          "port": "8051",
          "remoteIP": "http://localhost:31051"
        }
      ]
    }
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tunnel-send
  namespace: cnp-tunnel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tunnel-send
  template:
    metadata:
      labels:
        app: tunnel-send
    spec:
      containers:
      - name: tunnel-send
        image: nexus.cmss.com:8086/cnp/tunnel/send:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8050
          name: proxy-server-1
        - containerPort: 8051
          name: proxy-server-2
        volumeMounts:
        - mountPath: /opt/send/config.json
          name: tunnel-config
          subPath: config.json
      volumes:
      - name: tunnel-config
        configMap:
          name: tunnel-config
 
---
apiVersion: v1
kind: Service
metadata:
  name: tunnel-send
  namespace: cnp-tunnel
spec:
  type: NodePort
  ports:
  - protocol: TCP
    port: 8050
    targetPort: 8050
    nodePort: 31050
    name: proxy-server-1
  - protocol: TCP
    port: 8051
    targetPort: 8051
    nodePort: 31051
    name: proxy-server-2
  selector:
    app: tunnel-send
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack

至此，已经全部部署结束，执行下面的命令测试一下：

curl http://[集群A的中的宿主机IP]:31050/albums
curl http://[集群A的中的宿主机IP]:31051/albums

或者在浏览器里访问上面的地址，结果如下：

结束语

以上我们通过ssh隧道实现了跨集群的访问，目前只是在demo的程度，要在正式环境中使用的话，还需要考虑整个通信的稳定可靠的问题，例如给隧道增加心跳，多条隧道做负载均衡等。实际在业界还有例如Submariner（https://submariner.io/）等开源项目能轻松提供跨集群的安全应用访问，大家可以进一步学习了解。

本文参与腾讯云自媒体同步曝光计划，分享自微信公众号。

原始发表：2023-02-03，如有侵权请联系 cloudcommunity@tencent.com 删除

ssh