nginx+nginx-upsync-module实现动态负载及自定义验证

尚浩宇

发布于 2020-08-14 15:07:43

1.2K0

发布于 2020-08-14 15:07:43

文章被收录于专栏：杂烩杂烩

一、说明

nginx一般直接在配置文件里配置upstream即可实现负载均衡，但有些特定的环境下此种方式就显得有些局限性。比如后端服务器无法依据端口占用检查存活的时候；后台动态调整节点的时候；调整节点后不想修改配置文件重启nginx的时候等等。

此文的思路是将配置文件从nginx本地迁移到其他第三方服务上如etcd、consul上，然后时候拉取配置到本地。理论上说任何第三方配置中心都可以实现该功能，但需要对应的nginx模块。本文采用nginx-upsync-module，主要支持consul、etcd，本文以consul为例。

迁移配置文件还无法满足需求，还需要解决服务检测机制。这里不再以端口占用为准，而是实际访问某一个接口，查看是否返回数据，并以此为存活依据，最后通过调用consul的rest接口管理配置。

二、安装

2.1安装nginx

nginx安装参照前文《Linux下Nginx1.8安装》

需要注意的是，在安装nginx的时候需要安装nginx-upsync-module模块。

2.2安装nginx-upsync-module

打开https://github.com/weibocom/nginx-upsync-module，如果遇到github打不开，可以参照如下链接解决：https://www.php.cn/faq/445082.html

下载完成后，解压到linux目录备用。

2.3安装consul

consul的安装比较简单，这里不再赘述，可参照如下链接：https://blog.csdn.net/junaozun/article/details/90699384

三、搭建&测试

3.1搭建

本文的基础目录为：

1、nginx源码目录：/opt/server/software/nginx-1.19.1

2、nginx安装目录：/home/nginx/nginx

3、nginx-upsync-module目录：/opt/server/software/nginx-upsync-module/

确定consul正常运行后，配置nginx：

consul.conf

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    upstream testconsul {
        #这个配置无用了，但删除后启动会报错，如果consul里没有配置则会调用此地址，故此设置应该设置为一个默认的可用的地址，一旦从consul拉取到数据这个配置就无用了
        server 127.0.0.1:11111;
        #### 连接consul server，获取动态upstreams，配置负载均衡信息，间隔0.5s获取配置信息，upsync_timeout配置从consul拉取上游服务器配置的超时时间；upsync_interval配置从consul拉取上游服务器配置的间隔时间；upsync_type指定使用consul配置服务器；strong_dependency配置nginx在启动时是否强制依赖配置服务器，如果配置为on，则拉取配置失败时nginx启动同样失败
        upsync 192.168.1.97:8500/v1/kv/upstreams/testconsul/  upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
        ### 动态获取consul server相关负载均衡配置信息持久化在硬盘，这样即使consul服务器出问题了，本地还有一个备份。
        upsync_dump_path /home/nginx/nginx/conf/servers/testconsul.conf;
        }

        server {
                listen       80 ;
                server_name  testconsul;
                charset utf8;
                location /{
                        proxy_set_header Host $host;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_pass http://testconsul/HelloWord;

               }        

                access_log /home/nginx/nginx/logs/access-upsync.log;
                location /stub-status {
                        stub_status on;
                }       
                location = /upstream_show {
                    upstream_show;
                }   

          }     



    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

3.2测试

URL目录

1、负载状态：http://nginxhost:port/stub-status

2、当前负载节点列表：http://nginxhost:port/upstream_show

3、测试URL：http://nginxhost:port/HelloWord

4、consul增加节点（也可直接在UI上添加）：http://consulhost:port/v1/kv/upstreams/consultest/192.168.1.22:8080

四、自定义检测机制

核心功能是根据配置检测服务是否可用，然后根据结果更新到consul上。代码比较简单，主要代码如下：

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.cmgplex.aotucheck4consul</groupId>
	<artifactId>aotucheck4consul</artifactId>
	<version>1.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>aotucheck4consul</name>
	<url>http://www.cmgplex.com</url>

	<parent>
		<groupId>com.cmgplex.hr.parent</groupId>
		<artifactId>hr-parent</artifactId>
		<version>0.0.1-SNAPSHOT</version>
		<relativePath></relativePath>
	</parent>
	<properties>
		<snippetsDirectory>${project.build.directory}/generated-snippets</snippetsDirectory>
		<asciidoctor.input.directory>${project.basedir}/docs/asciidoc</asciidoctor.input.directory>
		<generated.asciidoc.directory>${project.build.directory}/asciidoc</generated.asciidoc.directory>
		<asciidoctor.html.output.directory>${project.build.directory}/asciidoc/html</asciidoctor.html.output.directory>
		<asciidoctor.pdf.output.directory>${project.build.directory}/asciidoc/pdf</asciidoctor.pdf.output.directory>
	</properties>

	<dependencies>
		<!--base dependency for all start -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter</artifactId>
			<exclusions>
				<exclusion>
					<groupId>org.springframework.boot</groupId>
					<artifactId>spring-boot-starter-logging</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>net.logstash.log4j</groupId>
			<artifactId>jsonevent-layout</artifactId>
			<version>1.0</version>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-api</artifactId>
		</dependency>
		<dependency>
			<groupId>com.google.guava</groupId>
			<artifactId>guava</artifactId>
		</dependency>
		<dependency>
			<groupId>commons-lang</groupId>
			<artifactId>commons-lang</artifactId>
		</dependency>
		<!--base dependency for all end -->

		<!--jetty begin -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
			<exclusions>
				<exclusion>
					<groupId>org.springframework.boot</groupId>
					<artifactId>spring-boot-starter-tomcat</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-undertow</artifactId>
		</dependency>
		<!--jetty end -->

		<!--consul begin -->
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-consul-all</artifactId>
		</dependency>
		<!--consul end -->

		<!--stream rabbit begin -->
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
		</dependency>
		<!--stream rabbit end -->
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
		</dependency>
		<!--feign begin -->
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-openfeign</artifactId>
		</dependency>
		<!--feign end -->

		<!--actuator begin -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<!--actuator end -->


		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.2.46</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-configuration-processor</artifactId>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>javax.mail</groupId>
			<artifactId>mail</artifactId>
			<version>1.4.7</version>
		</dependency>
		<dependency>
			<groupId>com.sun.mail</groupId>
			<artifactId>javax.mail</artifactId>
		</dependency>
		<!-- consul api begin -->
		<dependency>
			<groupId>com.ecwid.consul</groupId>
			<artifactId>consul-api</artifactId>
			<version>1.4.5</version>
		</dependency>
		<!-- consul api end -->
	</dependencies>
	<build>
		<finalName>aotucheck4consul</finalName>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<configuration>
					<source>${java.version}</source>
					<target>${java.version}</target>
					<encoding>UTF-8</encoding>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
		<resources>
			<resource>
				<directory>src/main/java</directory>
				<includes>
					<include>**/*.xml</include>
				</includes>
			</resource>
			<resource>
				<directory>src/main/resources</directory>
				<includes>
					<include>**/*.*</include>
				</includes>
			</resource>
			<resource>
				<directory>src/main/webapp</directory>
				<targetPath>META-INF/resources</targetPath>
				<includes>
					<include>**/*.*</include>
				</includes>
			</resource>
		</resources>
	</build>
	<!-- 这里是为了方便未配置maven的情况下，找到依赖 -->
	<repositories>
		<repository>
			<id>lumi-snapshots</id>
			<name>lumi-snapshots</name>
			<url>http://repo.lumiai.top/repository/maven-snapshots/</url>
			<layout>default</layout>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>
</project>

application.properties

spring.main.allow-bean-definition-overriding=true

#the following is for cloud

spring.datasource.hikari.idle-timeout = 600000
spring.datasource.hikari.connection-timeout = 30000
spring.datasource.hikari.max-lifetime = 1800000
spring.datasource.hikari.maximum-pool-size = 20
spring.datasource.type = com.zaxxer.hikari.HikariDataSource

#这个是关于mq的配置
spring.rabbitmq.addresses=amqp://192.168.1.2:5672
spring.rabbitmq.password=guest
spring.rabbitmq.username=guest
spring.cloud.stream.default-binder=rabbit
#定时检查，每隔1分钟
job.cleanConsulDeadService.cron=1 * * * * ? 
#consul配置key默认前缀
consul.default.key.prefix=upstreams/
#服务检查配置
#----------------api服务
#要检查服务的url，不包含地址
server.check.apps.api.url=/api/b
#要检查服务的host，只能是IP，包含端口，即使是80也不能少，不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.api.hosts=http://192.168.1.1:80,http://192.168.1.2:81
#要检查服务的method
server.check.apps.api.method=get
#暂不支持json参数
server.check.apps.api.param=get
#链接超时时间，单位毫秒
server.check.apps.api.timeout=3000
#断定服务正常的response返回code
server.check.apps.api.successcode=200,302
#----------------baidu服务
#要检查服务的url，不包含地址
server.check.apps.consultest.url=/s
#要检查服务的host，只能是IP，包含端口，即使是80也不能少，不然nginx那边读取不到配置会报错而导致配置不可用
server.check.apps.consultest.hosts=http://182.61.200.6:80
#要检查服务的method
server.check.apps.consultest.method=get
#暂不支持json参数
server.check.apps.consultest.param=wd=javahttp&tn=98012088_5_dg&ch=11
#链接超时时间，单位毫秒
server.check.apps.consultest.timeout=3000
#断定服务正常的response返回code
server.check.apps.consultest.successcode=200,302

#********************************************
#本地服务监听端口
server.port = 17000
#cpu的核数
server.undertow.io-threads=4
#预估的最佳线程数
server.undertow.worker-threads=400
spring.application.name=aotucheck4consul
spring.profiles.active=dev
#********************************************
#consul的配置
spring.cloud.consul.enabled=true
spring.cloud.consul.host=192.168.1.2
spring.cloud.consul.port=8500

#********************************************
#注册中心的配置
spring.cloud.consul.discovery.enabled=false
spring.cloud.consul.discovery.prefer-ip-address=true
spring.cloud.consul.discovery.health-check-path=/actuator/health
spring.cloud.consul.discovery.health-check-interval=10s
spring.cloud.consul.discovery.health-check-timeout=1s
spring.cloud.consul.discovery.instance-id=${spring.application.name}:${spring.cloud.client.ip-address}:${server.port}


#是否把自己向注册中心注册，可以做纯consumer不注册自己
spring.cloud.consul.discovery.register=true

#********************************************
#配置服务的配置
spring.cloud.consul.config.enabled=true
spring.cloud.consul.config.prefix=config
spring.cloud.consul.config.profile-separator=,
spring.cloud.consul.config.default-context=application
spring.cloud.consul.config.watch.enabled=true
spring.cloud.consul.config.watch.delay=1000
spring.cloud.consul.config.watch.wait-time=3
spring.cloud.consul.config.format=properties
spring.cloud.consul.config.data-key=configuration
#禁用熔断器首次调用时强制1秒超时
hystrix.command.default.execution.timeout.enabled=false

CheckServiceScheduler.java

@Component
@EnableScheduling
public class CheckServiceScheduler {
	private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceScheduler.class);
	@Autowired
	private ICheckService checkService;

	
	@Scheduled(cron = "${job.cleanConsulDeadService.cron}")
	public void checkServer() {
		LOGGER.info("-----------------------------start checkServer Scheduler-----------------------------");
		this.checkService.check();
		LOGGER.info("-----------------------------end checkServer Scheduler-----------------------------");
	}
}

CheckServiceImpl.java

@Service
public class CheckServiceImpl implements ICheckService {
	private static final Logger LOGGER = LoggerFactory.getLogger(CheckServiceImpl.class);
	@Resource
	private ServerPropertiesConfig serverPropertiesConfig;
	@Resource
	private IConsulService consulService;
	@Value("${consul.default.key.prefix}")
	private String consulKeyPrefix;

	/*
	 * (非 Javadoc) <p>Title: check</p> <p>Description: </p>
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.ICheckService#check()
	 */
	@Override
	public void check() {
		LOGGER.debug("serverPropertiesConfig={}", this.serverPropertiesConfig);
		Map<String, ServerInfoVo> apps = this.serverPropertiesConfig.getApps();
		for (Entry<String, ServerInfoVo> app : apps.entrySet()) {
			String appName = app.getKey();
			LOGGER.info("appName={}", appName);
			ServerInfoVo config = app.getValue();
			LOGGER.info("config={}", config);
			List<Integer> expectCodeList = ListUtils.str2list(config.getSuccesscode(), ",");
			List<String> hostsList = ListUtils.str2list4String(config.getHosts(), ",");
			String method = config.getMethod();
			HttpMethod httpMethod = HttpMethod.valueOf(method.toUpperCase());
			String param = config.getParam();
			Integer timeout = config.getTimeout();
			hostsList.parallelStream().forEach(host -> {
				try {
					LOGGER.info("host={}", host);
					String httpurl = host + config.getUrl();
					LOGGER.info("httpurl={}", httpurl);
					boolean isSuccess = HttpTools.compareCode(expectCodeList, httpMethod, httpurl, param, timeout);
					LOGGER.info("isSuccess={}", isSuccess);
					String key = this.consulKeyPrefix + appName + "/"
							+ host.replace("http://", "").replace("https://", "");
					LOGGER.info("key={}", key);
					// 如果服务可用
					if (isSuccess) {
						// 检查consul上配置是否正常
						if (!this.consulService.isKeyExist(key)) {
							// 如果不正常，则更新正常
							this.consulService.addKeyValue(key, new NginxServerConfig(2, 1, 10));
							LOGGER.info("server success,consul faild,now update consul,key={}", key);
						}
					} else {
						// 如果服务不可用
						// 检查consul上配置是否正常
						if (this.consulService.isKeyExist(key)) {
							// 如果正常，则更新为不正常
							this.consulService.deleteKey(key);
							LOGGER.info("server faild,consul success,now update consul,key={}", key);
						}
					}
				} catch (Exception e) {
					LOGGER.error(e.getMessage(), e);
				}
			});
		}
	}

}

ConsulServiceImpl.java

@Service
public class ConsulServiceImpl implements IConsulService {
	private static final Logger LOGGER = LoggerFactory.getLogger(ConsulServiceImpl.class);
	private static final BASE64Decoder decoder = new BASE64Decoder();
	@Autowired
	private ConsulClient consulClient;

	/*
	 * (非 Javadoc) <p>Title: getKeysByPrefix</p> <p>Description: </p>
	 * 
	 * @param keyPrefix
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getKeysByPrefix(java.lang.String)
	 */
	@Override
	public List<String> getKeysByPrefix(String keyPrefix) {
		Response<List<GetValue>> valus = this.consulClient.getKVValues(keyPrefix);
		return valus.getValue().stream().map(value -> value.getKey()).filter(key -> StringUtils.isNotBlank(key))
				.collect(Collectors.toList());
	}

	/*
	 * (非 Javadoc) <p>Title: getValue</p> <p>Description: </p>
	 * 
	 * @param key
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#getValue(java.lang.String)
	 */
	@Override
	public NginxServerConfig getValue(String key) {
		GetValue kvValue = this.consulClient.getKVValue(key).getValue();
		if (null == kvValue) {
			return null;
		}
		String value = kvValue.getValue();
		String jsonValue;
		try {
			jsonValue = new String(decoder.decodeBuffer(value), "UTF-8");
			return JSON.parseObject(jsonValue, NginxServerConfig.class);
		} catch (IOException e) {
			LOGGER.error(e.getMessage(), e);
		}
		return null;
	}

	/*
	 * (非 Javadoc) <p>Title: isKeyExist</p> <p>Description: </p>
	 * 
	 * @param key
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#isKeyExist(java.lang.String)
	 */
	@Override
	public boolean isKeyExist(String key) {
		return null != this.getValue(key);
	}

	/*
	 * (非 Javadoc) <p>Title: deleteKey</p> <p>Description: </p>
	 * 
	 * @param key
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#deleteKey(java.lang.String)
	 */
	@Override
	public boolean deleteKey(String key) {
		return null != this.consulClient.deleteKVValue(key);
	}

	/*
	 * (非 Javadoc) <p>Title: addKeyValue</p> <p>Description: </p>
	 * 
	 * @param key
	 * 
	 * @param value
	 * 
	 * @return
	 * 
	 * @see com.cmgplex.aotucheck4consul.jobsystem.service.IConsulService#addKeyValue(java.lang.String,
	 * com.cmgplex.aotucheck4consul.jobsystem.vo.NginxServerConfig)
	 */
	@Override
	public boolean addKeyValue(String key, NginxServerConfig value) {
		return this.consulClient.setKVValue(key, JSON.toJSONString(value)).getValue();
	}

}

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

如有侵权请联系 cloudcommunity@tencent.com 删除

nginx