适用于事件响应运行手册的生产就绪模板，涵盖检测、分类、缓解、解决和沟通。

SEV1 Complete outage, data loss 15 min Production down

SEV2 Major degradation 30 min Critical feature broken

SEV3 Minor impact 2 hours Non-critical bug

SEV4 Minimal impact Next business day Cosmetic issue

1. Overview & Impact
2. Detection & Alerts
3. Initial Triage
4. Mitigation Steps
5. Root Cause Investigation
6. Resolution Procedures
7. Verification & Rollback
8. Communication Templates
9. Escalation Matrix

# [Service Name] Outage Runbook

## Overview
**Service**: Payment Processing Service
**Owner**: Platform Team
**Slack**: #payments-incidents
**PagerDuty**: payments-oncall

## Impact Assessment
- [ ] Which customers are affected?
- [ ] What percentage of traffic is impacted?
- [ ] Are there financial implications?
- [ ] What's the blast radius?

## Detection
### Alerts
- `payment_error_rate > 5%` (PagerDuty)
- `payment_latency_p99 > 2s` (Slack)
- `payment_success_rate < 95%` (PagerDuty)

### Dashboards
- [Payment Service Dashboard](https://grafana/d/payments)
- [Error Tracking](https://sentry.io/payments)
- [Dependency Status](https://status.stripe.com)

## Initial Triage (First 5 Minutes)

### 1. Assess Scope

kubectl 获取 pods -n payments -l app=payment-service

kubectl rollout history deployment/payment-service -n payments

curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))"


### 2. Quick Health Checks
- [ ] Can you reach the service? `curl -I https://api.company.com/payments/health`
- [ ] Database connectivity? Check connection pool metrics
- [ ] External dependencies? Check Stripe, bank API status
- [ ] Recent changes? Check deploy history

### 3. Initial Classification
| Symptom | Likely Cause | Go To Section |
|---------|--------------|---------------|
| All requests failing | Service down | Section 4.1 |
| High latency | Database/dependency | Section 4.2 |
| Partial failures | Code bug | Section 4.3 |
| Spike in errors | Traffic surge | Section 4.4 |

## Mitigation Procedures

### 4.1 Service Completely Down

第2步：如果 pods 处于崩溃循环，检查日志

kubectl logs -n payments -l app=payment-service --tail=100

kubectl 回滚历史部署/payment-service -n payments

第4步：如果最近的部署可疑，则回滚

kubectl 回滚 deployment/payment-service -n payments

步骤 5：如果资源有限，则扩大规模

kubectl scale deployment/payment-service -n payments --replicas=10

kubectl rollout status deployment/payment-service -n payments

kubectl exec -n payments deploy/payment-service -- "}

步骤 2：检查慢查询（如果是数据库问题）

SELECT pid, now() - query_start AS duration, query

WHERE state = 'active' AND duration > interval '5 seconds'

步骤3：如有需要，终止长时间运行的查询

psql -h $DB_HOST -U $DB_USER -c "SELECT pg_terminate_backend(pid);"

curl -w "@curl-format.txt" -o /dev/null -s https://api.stripe.com/v1/health

步骤5：如果依赖慢，则启用断路器

kubectl set env deployment/payment-service "}

STRIPE_CIRCUIT_BREAKER_ENABLED=true -n payments


### 4.3 Partial Failures (Specific Errors)

kubectl logs -n payments -l app=payment-service --tail=500 "

grep -i error sort uniq -c sort -rn head -20

访问 Sentry: https://sentry.io/payments

第3步：如果是特定端点，启用功能开关以禁用

curl -X POST https://api.company.com/internal/feature-flags "}

-d '{"flag": "DISABLE_PROBLEMATIC_FEATURE", "enabled": true}'

步骤 4：如果存在数据问题，检查最近的数据更改

并且 created_at > now() - interval '1 hour';

kubectl scale deployment/payment-service -n payments --replicas=20

步骤 4：如果遭到攻击，阻止可疑的 IP

Incident Runbook Templates 实用指南

Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.

大数据

事件响应运行手册生产就绪模板，覆盖检测、分类、缓解、解决全流程，提供服务中断与数据库故障的详细操作命令、升级路径及沟通模板，助力团队快速应对SEV1-4级事件，提升待命工程师响应效率。

Kubernetes

数据库

Agent

2026上云采购 | AI焕新·智启新局

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

EdgeOne AI 安全实战专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

Incident Runbook Templates 实用指南

Incident Runbook Templates 实用指南

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐