前提:
要使用 Spring Boot 实现简单的 ELK(Elasticsearch、Logstash、Kibana)系统,需要满足一系列前提条件,涵盖环境准备、技术基础、组件认知等多个方面。以下是详细的前提说明:
实现 ELK 系统的首要前提是搭建好运行所需的基础环境,确保各组件能正常启动和通信。
JAVA_HOME
环境变量,确保命令行可识别java
和javac
命令。vm.max_map_count=262144
)。需单独安装 Elasticsearch、Logstash、Kibana,并完成基础配置(以单机版为例,集群版需额外配置)。
Elasticsearch
作用:存储和索引日志数据。
安装:从官网下载对应版本,解压后即可运行(bin/elasticsearch
)。
基础配置(config/elasticsearch.yml
):
yaml
cluster.name: my-elk-cluster # 集群名称(单机可自定义)
node.name: node-1 # 节点名称
network.host: 0.0.0.0 # 允许所有IP访问(开发环境)
http.port: 9200 # HTTP端口
验证:访问http://localhost:9200
,返回节点信息即启动成功。如下:
Logstash
作用:收集、过滤、转换日志数据,发送到 Elasticsearch。
安装:从官网下载,解压后配置管道(config/logstash-simple.conf
):
conf
input {
tcp {
port => 5000 # 接收Spring Boot日志的端口
codec => json_lines # 解析JSON格式日志
}
}
output {
elasticsearch {
hosts => ["localhost:9200"] # Elasticsearch地址
index => "springboot-logs-%{+YYYY.MM.dd}" # 日志索引名(按天分割)
}
stdout { codec => rubydebug } # 同时输出到控制台(调试用)
}
启动:bin/logstash -f config/logstash-simple.conf
。如下:
Kibana
作用:可视化展示 Elasticsearch 中的日志数据。
安装:从官网下载,解压后配置(config/kibana.yml
):
yaml
server.host: "0.0.0.0" # 允许所有IP访问
elasticsearch.hosts: ["http://localhost:9200"] # 连接Elasticsearch
启动:bin/kibana
,访问http://localhost:5601
进入控制台。如下:
需开发或改造 Spring Boot 应用,使其能生成结构化日志并发送到 Logstash。
项目基础
logback
或log4j2
)。logstash-logback-encoder
)。日志配置
目标:将 Spring Boot 日志以JSON 格式通过 TCP 发送到 Logstash 的 5000 端口(与 Logstash 输入配置对应)。
以logback
为例,在src/main/resources
下创建logback-spring.xml
:
xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>localhost:5000</destination> <!-- Logstash地址和端口 -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<!-- 自定义字段(可选) -->
<includeMdcKeyName>requestId</includeMdcKeyName>
<customFields>{"application":"my-springboot-app"}</customFields>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="LOGSTASH" />
<appender-ref ref="CONSOLE" /> <!-- 同时输出到控制台 -->
</root>
</configuration>
依赖:在pom.xml
中添加 Logstash 编码器(若使用 logback):
xml
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>7.4.0</version>
</dependency>
telnet
或nc
测试端口连通性(如检查 Spring Boot 到 Logstash 的 5000 端口是否可通)。logs/elasticsearch.log
logs/logstash-plain.log
logs/kibana.log
在springboot的配置文件中编写访问地址:
spring.application.name=elkdemo
logName= #日志的名称catalina-2025.07.30
elasticsearchHost= #es的地址
elasticsearchPort= #es的端口号9200
elasticsearchDefaultHost= #默认的es地址localhost:9200
编写ES的config配置类
package com.example.demo.config;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate;
import co.elastic.clients.elasticsearch.ElasticsearchClient;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.elc.ElasticsearchClients;
@Configuration
public class ElasticsearchConfig {
@Value("${elasticsearchHost}")
private String elasticsearchHost;
@Value("${elasticsearchPort}")
private Integer elasticsearchPort;
@Value("${elasticsearchDefaultHost}")
private String elasticsearchDefaultHost;
@Bean
public RestHighLevelClient restHighLevelClient() {
// 配置Elasticsearch地址
return new RestHighLevelClient(
RestClient.builder(
new HttpHost(elasticsearchHost, elasticsearchPort, "http")
)
);
}
@Bean
public ElasticsearchClient elasticsearchClient() {
// 使用相同的连接配置创建ElasticsearchClient
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(elasticsearchDefaultHost)
.build();
return ElasticsearchClients.createImperative(clientConfiguration);
}
@Bean
public ElasticsearchTemplate elasticsearchTemplate() {
return new ElasticsearchTemplate(elasticsearchClient());
}
}
编写两个基础的controller接口
package com.example.demo.controller;
import com.example.demo.model.Document;
import com.example.demo.service.SearchService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import java.io.IOException;
import java.util.List;
import java.util.Map;
@RestController
@RequestMapping("/api")
public class SearchController {
@Autowired
private SearchService searchService;
// 搜索接口
@GetMapping("/search")
public List<Document> search(
//query就是要搜索的关键字
@RequestParam String query,
@RequestParam(defaultValue = "1") int page,
@RequestParam(defaultValue = "10") int size
) throws IOException {
return searchService.searchDocuments(query, page, size);
}
// 详情接口
@GetMapping("/document/{id}")
public Map<String, Object> getDocumentDetail(
@PathVariable String id,
@RequestParam String indexName){
Map<String, Object> documentById = searchService.getDocumentById(id, indexName);
return documentById;
}
}
编写对应的实现类
package com.example.demo.service;
import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch.core.GetResponse;
import com.example.demo.model.Document;
import co.elastic.clients.elasticsearch.core.GetRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.MultiMatchQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.search.sort.SortOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@Service
public class SearchService {
@Autowired
private RestHighLevelClient client;
@Value("${logName}")
private String logName;
@Autowired
private ElasticsearchClient elasticsearchClient;
public List<Document> searchDocuments(String query, int page, int size) throws IOException {
// 使用存在的索引名(在配置文件编写)
SearchRequest searchRequest = new SearchRequest(logName);
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
// 只搜索映射中存在的字段
MultiMatchQueryBuilder multiMatchQuery = QueryBuilders.multiMatchQuery(
query,
"@version",
"event.original", // 嵌套字段
"host.name",
"log.file.path",
"message",
"tags"
);
sourceBuilder.query(multiMatchQuery);
//分页开始位置
sourceBuilder.from((page - 1) * size);
//每一页的大小
sourceBuilder.size(size);
//按照时间降序排序
sourceBuilder.sort(SortBuilders.fieldSort("@timestamp").order(SortOrder.DESC));
//执行搜索
searchRequest.source(sourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
List<Document> documents = new ArrayList<>();
//遍历es中命中的文档
for (SearchHit hit : searchResponse.getHits()) {
//获取到的源数据进行类型转换为map对象
Map<String, Object> source = hit.getSourceAsMap();
Document document = new Document();
document.setId(hit.getId());
//使用 @timestamp 作为标题(时间戳)
document.setTitle((String) source.get("@timestamp"));
//处理嵌套字段 event
Map<String, Object> event = (Map<String, Object>) source.get("event");
if (event != null) {
document.setContent((String) event.get("original"));
}
document.setTimestamp((String) source.get("@timestamp"));
documents.add(document);
}
return documents;
}
public Map<String,Object> getDocumentById(String id, String indexName) {
try {
GetRequest request = new GetRequest.Builder()
.index(indexName)
.id(id)
.build();
//转换
GetResponse<Map> response = elasticsearchClient.get(request, Map.class);
if (response.found()) {
return response.source(); // 返回完整文档内容
} else {
throw new RuntimeException("文档不存在: " + id + " in index " + indexName);
}
} catch (IOException e) {
throw new RuntimeException("查询失败", e);
}
}
}
编写Modle实体类
package com.example.demo.model;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
public class Document {
@Id
private String id;
@Field(type = FieldType.Text)
private String title;
@Field(type = FieldType.Text)
private String content;
@Field(type = FieldType.Date)
private String timestamp;
public String getId() { return id; }
public void setId(String id) { this.id = id; }
public String getTitle() { return title; }
public void setTitle(String title) { this.title = title; }
public String getContent() { return content; }
public void setContent(String content) { this.content = content; }
public String getTimestamp() { return timestamp; }
public void setTimestamp(String timestamp) { this.timestamp = timestamp; }
}
在resource目录下编写简单的前端代码index.html
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>ELK 日志搜索系统</title>
<script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
<link href="https://cdn.jsdelivr.net/npm/font-awesome@4.7.0/css/font-awesome.min.css" rel="stylesheet">
<style>
* {
box-sizing: border-box;
}
body {
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
margin: 0;
padding: 20px;
background-color: #f5f5f5;
}
.container {
max-width: 1200px;
margin: 0 auto;
}
.search-box {
background-color: white;
padding: 20px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
margin-bottom: 20px;
}
.search-input {
width: 80%;
padding: 10px;
font-size: 16px;
border: 1px solid #ddd;
border-radius: 4px;
margin-right: 10px;
}
.search-button {
padding: 10px 20px;
background-color: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
font-size: 16px;
}
.search-button:hover {
background-color: #0b7dda;
}
.result-list {
background-color: white;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
padding: 20px;
}
.result-item {
border-bottom: 1px solid #eee;
padding: 15px 0;
cursor: pointer;
}
.result-item:last-child {
border-bottom: none;
}
.result-item:hover {
background-color: #f9f9f9;
}
.result-title {
font-size: 18px;
color: #2196F3;
margin-bottom: 5px;
}
.result-meta {
font-size: 14px;
color: #666;
margin-bottom: 10px;
}
.result-content {
font-size: 15px;
color: #333;
line-height: 1.5;
max-height: 60px;
overflow: hidden;
text-overflow: ellipsis;
}
.pagination {
margin-top: 20px;
display: flex;
justify-content: center;
}
.page-button {
padding: 8px 16px;
margin: 0 5px;
border: 1px solid #ddd;
border-radius: 4px;
cursor: pointer;
}
.page-button.active {
background-color: #2196F3;
color: white;
border-color: #2196F3;
}
.no-results {
text-align: center;
padding: 50px 0;
color: #666;
}
</style>
</head>
<body>
<div class="container">
<div class="search-box">
<h2>ELK 日志搜索系统</h2>
<div>
<input type="text" id="query" class="search-input" placeholder="请输入搜索关键词...">
<button class="search-button" onclick="search()">
<i class="fa fa-search"></i> 搜索
</button>
</div>
<div style="margin-top: 10px; font-size: 14px; color: #666;">
支持关键词搜索,例如: <code>ERROR</code>、<code>command line</code>、<code>2025-07-30</code>
</div>
</div>
<div class="result-list" id="results">
<div class="no-results">请输入关键词进行搜索</div>
</div>
<div class="pagination" id="pagination">
<!-- 分页按钮将动态生成 -->
</div>
</div>
<script>
// 当前页码和每页大小
let currentPage = 1;
const pageSize = 10;
let totalPages = 1;
let currentQuery = '';
// 搜索函数
async function search(page = 1) {
const queryInput = document.getElementById('query');
currentQuery = queryInput.value.trim();
currentPage = page;
if (!currentQuery) {
alert('请输入搜索关键词');
return;
}
try {
// 显示加载状态
document.getElementById('results').innerHTML = '<div class="no-results"><i class="fa fa-spinner fa-spin"></i> 正在搜索...</div>';
const response = await axios.get('/api/search', {
params: {
query: currentQuery,
page: currentPage,
size: pageSize
}
});
renderResults(response.data);
renderPagination();
} catch (error) {
console.error('搜索失败:', error);
document.getElementById('results').innerHTML = '<div class="no-results"><i class="fa fa-exclamation-triangle"></i> 搜索失败,请重试</div>';
}
}
// 渲染搜索结果
function renderResults(documents) {
const resultsDiv = document.getElementById('results');
if (!documents || documents.length === 0) {
resultsDiv.innerHTML = '<div class="no-results"><i class="fa fa-search"></i> 没有找到匹配的结果</div>';
return;
}
const resultItems = documents.map(doc => `
<div class="result-item" onclick="openDetail('${doc.id}', 'catalina-2025.07.30')">
<div class="result-title">${doc.title || '无标题'}</div>
<div class="result-meta">
<span><i class="fa fa-clock-o"></i> ${doc.timestamp || '未知时间'}</span>
<span style="margin-left: 15px;"><i class="fa fa-file-text-o"></i> ${doc.id}</span>
</div>
<div class="result-content">${doc.content ? doc.content.substr(0, 200) + '...' : '无内容'}</div>
</div>
`).join('');
resultsDiv.innerHTML = resultItems;
}
// 渲染分页控件
function renderPagination() {
const paginationDiv = document.getElementById('pagination');
// 假设后端返回总页数
// 实际应用中应从后端获取总记录数,计算总页数
totalPages = Math.ceil(50 / pageSize); // 示例:假设总共有50条记录
let paginationHtml = '';
// 上一页按钮
if (currentPage > 1) {
paginationHtml += `<button class="page-button" onclick="search(${currentPage - 1})">上一页</button>`;
}
// 页码按钮
const maxVisiblePages = 5;
let startPage = Math.max(1, currentPage - Math.floor(maxVisiblePages / 2));
let endPage = Math.min(startPage + maxVisiblePages - 1, totalPages);
if (endPage - startPage + 1 < maxVisiblePages) {
startPage = Math.max(1, endPage - maxVisiblePages + 1);
}
for (let i = startPage; i <= endPage; i++) {
paginationHtml += `<button class="page-button ${i === currentPage ? 'active' : ''}" onclick="search(${i})">${i}</button>`;
}
// 下一页按钮
if (currentPage < totalPages) {
paginationHtml += `<button class="page-button" onclick="search(${currentPage + 1})">下一页</button>`;
}
paginationDiv.innerHTML = paginationHtml;
}
// 打开详情页
function openDetail(id, indexName) {
window.location.href = `detail.html?id=${id}&index=${indexName}`;
}
</script>
</body>
</html>
在resource目录下编写简单的前端代码detail.html
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>日志详情 | ELK 搜索系统</title>
<script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
<link href="https://cdn.jsdelivr.net/npm/font-awesome@4.7.0/css/font-awesome.min.css" rel="stylesheet">
<style>
* {
box-sizing: border-box;
margin: 0;
padding: 0;
}
body {
font-family: 'Consolas', 'Monaco', monospace;
background-color: #f5f5f5;
padding: 20px;
line-height: 1.5;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
border-radius: 8px;
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
padding: 20px;
}
.header {
margin-bottom: 20px;
}
.back-button {
padding: 8px 16px;
background-color: #2196F3;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
display: inline-flex;
align-items: center;
gap: 8px;
margin-bottom: 15px;
}
.back-button:hover {
background-color: #0b7dda;
}
.meta-info {
margin-bottom: 20px;
padding: 10px;
background-color: #f9f9f9;
border-radius: 4px;
font-size: 14px;
}
.meta-item {
margin-right: 20px;
display: inline-block;
}
.json-container {
background-color: #f9f9f9;
border-radius: 4px;
padding: 20px;
overflow-x: auto;
white-space: pre-wrap;
}
.json-key {
color: #0033a0;
font-weight: bold;
}
.json-string {
color: #008000;
}
.json-number {
color: #800000;
}
.json-boolean {
color: #0000ff;
}
.json-null {
color: #808080;
}
.error {
color: #dc3545;
padding: 20px;
text-align: center;
background-color: #f8d7da;
border-radius: 4px;
}
.loading {
text-align: center;
padding: 50px 0;
color: #666;
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<button class="back-button" onclick="goBack()">
<i class="fa fa-arrow-left"></i> 返回搜索结果
</button>
<div class="meta-info">
<div class="meta-item">
<i class="fa fa-database"></i> <span id="index-name">加载中...</span>
</div>
<div class="meta-item">
<i class="fa fa-file-text-o"></i> <span id="document-id">加载中...</span>
</div>
<div class="meta-item">
<i class="fa fa-clock-o"></i> <span id="load-time">加载中...</span>
</div>
</div>
</div>
<div id="loading" class="loading">
<i class="fa fa-spinner fa-spin"></i> 正在加载数据...
</div>
<div id="error" class="error" style="display: none;"></div>
<div id="json-container" class="json-container" style="display: none;"></div>
</div>
<script>
// 原生JSON高亮格式化函数
function syntaxHighlight(json) {
if (typeof json !== 'string') {
json = JSON.stringify(json, undefined, 2);
}
// 正则匹配不同JSON元素并添加样式类
json = json.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');
return json.replace(/("(\\u[a-zA-Z0-9]{4}|\\[^u]|[^\\"])*"(\s*:)?|\b(true|false|null)\b|-?\d+(?:\.\d*)?(?:[eE][+-]?\d+)?)/g, function (match) {
let cls = 'json-number';
if (/^"/.test(match)) {
if (/:$/.test(match)) {
cls = 'json-key';
} else {
cls = 'json-string';
}
} else if (/true|false/.test(match)) {
cls = 'json-boolean';
} else if (/null/.test(match)) {
cls = 'json-null';
}
return '<span class="' + cls + '">' + match + '</span>';
});
}
// 页面加载完成后执行
document.addEventListener('DOMContentLoaded', function() {
// 获取URL参数
const urlParams = new URLSearchParams(window.location.search);
const docId = urlParams.get('id');
const indexName = urlParams.get('index');
// 验证参数
if (!docId || !indexName) {
document.getElementById('loading').style.display = 'none';
document.getElementById('error').textContent = '错误:缺少文档ID或索引名参数';
document.getElementById('error').style.display = 'block';
return;
}
// 更新元信息
document.getElementById('document-id').textContent = `文档ID: ${docId}`;
document.getElementById('index-name').textContent = `索引: ${indexName}`;
// 记录开始时间
const startTime = Date.now();
// 请求数据
axios.get(`/api/document/${docId}`, {
params: {
indexName: indexName
},
timeout: 15000
})
.then(response => {
// 计算加载时间
const loadTime = Date.now() - startTime;
document.getElementById('load-time').textContent = `加载时间: ${loadTime}ms`;
// 隐藏加载状态,显示内容
document.getElementById('loading').style.display = 'none';
document.getElementById('json-container').style.display = 'block';
// 格式化并显示JSON
document.getElementById('json-container').innerHTML = syntaxHighlight(response.data);
})
.catch(error => {
// 处理错误
document.getElementById('loading').style.display = 'none';
let errorMsg = '加载失败: ';
if (error.response) {
errorMsg += `服务器返回 ${error.response.status} 错误`;
} else if (error.request) {
errorMsg += '未收到服务器响应,请检查网络';
} else {
errorMsg += error.message;
}
document.getElementById('error').textContent = errorMsg;
document.getElementById('error').style.display = 'block';
});
});
// 返回上一页
function goBack() {
window.history.back();
}
</script>
</body>
</html>
访问localhost:8080即可展示界面,如下:
当我们搜索某个关键字时,是支持全文索引的:
当点击某个具体的文档时,可以查看详情:
config/jvm.options
中的-Xms2g -Xmx2g
)。