20K Java+AI 岗位高频考点 — 可观测性(Observability)是现代运维的核心能力。覆盖Metrics/Tracing/Logs三大支柱、Prometheus+Grafana监控体系、OpenTelemetry、链路追踪、告警设计、SLI/SLO等。
Metrics(指标):
Tracing(链路追踪):
Logs(日志):
关系:Metrics发现问题 → Tracing定位到具体调用 → Logs查看详细上下文。
示例:
SLI: 订单API的P99响应时间
SLO: P99 < 300ms(月度)
SLA: 月度SLO低于99.95%时赔偿10%月费
Google SRE提出的四个关键监控指标:
1. Latency(延迟):请求响应时间(区分成功和失败请求)
2. Traffic(流量):请求量、QPS
3. Errors(错误):错误率、错误类型
4. Saturation(饱和度):资源使用率(CPU、内存、连接池、队列深度)
Brendan Gregg提出的资源分析框架:
适用:CPU、内存、磁盘、网络、连接池等所有资源。
Tom Wilkie提出的微服务监控方法(USE的补充):
Application(Exporter/Micrometer) ← scrape ← Prometheus Server
↓
TSDB(本地存储) + Alertmanager
↓
Grafana(Dashboard)
核心组件:
// Spring Boot Micrometer
@Autowired MeterRegistry registry;
// Counter
Counter.builder("orders.created")
.tag("type", "normal")
.register(registry).increment();
// Gauge
AtomicInteger activeConnections = registry.gauge("connections.active",
new AtomicInteger(0));
// Timer(Histogram)
Timer.Sample sample = Timer.start(registry);
doSomething();
sample.stop(Timer.builder("orders.processing.time")
.tag("status", "success")
.register(registry));
# QPS(每秒请求速率)
rate(http_server_requests_seconds_count[5m])
# P99延迟
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
# 错误率
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
# Top N 最慢接口
topk(10, histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m])))
# 容量预测(线性回归)
predict_linear(disk_usage_bytes[1h], 3600)
# 内存使用率
jvm_memory_used_bytes / jvm_memory_max_bytes
静态配置:
scrape_configs:
- job_name: 'my-app'
static_configs:
- targets: ['localhost:8080']
Kubernetes SD(自动发现K8s中的Pod):
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
Nacos SD:通过Nacos注册中心自动发现服务。
方案一:HA Pair:两个独立的Prometheus采集相同目标
方案二:Thanos:长期存储 + 全局查询 + 高可用
方案三:Cortex:多租户、长期存储
方案四:VictoriaMetrics:高性能替代方案
Prometheus-1 → Thanos Sidecar → Thanos Store Gateway → Thanos Query → Grafana
Prometheus-2 → Thanos Sidecar ↗ ↑
Thanos Query Frontend ←←←←
层级结构:
1. Overview Dashboard:全局视角,关键指标概览
2. Service Dashboard:单个服务的详细指标
3. Infrastructure Dashboard:基础设施(CPU/内存/磁盘/网络)
4. Business Dashboard:业务指标(订单量/转化率/收入)
设计原则:
{
"name": "service",
"type": "query",
"query": "label_values(http_server_requests_seconds_count, application)"
}
在Dashboard中使用:application="$service"
常用变量类型:
query:从Prometheus查询interval:时间间隔custom:自定义选项datasource:数据源切换
# Grafana Alert Rule
apiVersion: 1
groups:
- name: service-alerts
rules:
- uid: alert-1
title: High Error Rate
condition: C
data:
- refId: A
relativeTimeRange:
from: 600 # 10分钟
datasourceUid: prometheus
model:
expr: sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m]))
- refId: C
reducer: last
noDataState: OK
execErrState: Alerting
for: 5m
Grafana支持在同一Dashboard中混合不同数据源:
联动查询:从Metrics发现异常 → 点击跳转到Logs → 从Logs跳转到Traces。
传播方式:HTTP Header / gRPC Metadata
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
OpenTelemetry(OTel)是CNCF的可观测性标准,统一了Traces、Metrics、Logs的采集和传播。
核心组件:
Spring Boot集成:
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>
// 自动采集HTTP请求、JDBC调用、Redis调用
// 手动创建Span
@Autowired Tracer tracer;
Span span = tracer.spanBuilder("process-order")
.setAttribute("order.id", orderId)
.startSpan();
try {
doProcess(orderId);
} finally {
span.end();
}
| 维度 | SkyWalking | Jaeger | Zipkin |
|---|---|---|---|
| 语言 | Java | Go | Java/Scala |
| 采样 | 支持多种策略 | 支持多种 | 有限 |
| UI | 丰富 | 一般 | 简洁 |
| 告警 | 内置 | 无 | 无 |
| Java Agent | 强(字节码增强) | 需手动埋点 | 需手动埋点 |
| 存储后端 | ES/MySQL/H2 | ES/Cassandra | ES/MySQL |
| 协议 | 自定义/Sw | OpenTelemetry | Zipkin |
选型建议:Java项目推荐SkyWalking(无侵入Agent),云原生推荐Jaeger+OTel。
# OpenTelemetry Collector 尾部采样
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors
type: status_code
status_code:
status_codes:
- ERROR
- name: slow
type: latency
latency:
threshold_ms: 5000
- name: sample
type: probabilistic
probabilistic:
sampling_percentage: 10
应用 → Logback/Log4j → Filebeat → Logstash → Elasticsearch → Kibana
↓
Grafana(可选)
替代方案:
<!-- logback-spring.xml -->
<configuration>
<springProfile name="prod">
<!-- JSON格式(方便ES解析) -->
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeMdc>true</includeMdc>
<customFields>{"app":"order-service"}</customFields>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="JSON"/>
</root>
</springProfile>
</configuration>
日志规范:
// MDC注入TraceId
@Component
public class TraceFilter implements Filter {
@Override
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) {
String traceId = extractTraceId((HttpServletRequest) req);
MDC.put("traceId", traceId);
try {
chain.doFilter(req, res);
} finally {
MDC.clear();
}
}
}
Logback配置:
<pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] [%X{traceId}] %-5level %logger - %msg%n</pattern>
这样在Kibana/Loki中可以通过traceId关联日志和链路。
| 维度 | Loki | Elasticsearch |
|---|---|---|
| 存储 | 只索引标签,不索引日志正文 | 全文索引 |
| 成本 | 低(对象存储) | 高(ES集群) |
| 查询 | LogQL(标签过滤+正则) | Lucene DSL(全文搜索) |
| 适用 | 结构化标签日志 | 全文搜索 |
| 部署 | 简单 | 复杂 |
建议:如果只需要按标签查日志,Loki更轻量;需要全文搜索,用ES。
1. 可操作性:每条告警必须有明确的处理动作
2. 分级:P0(电话)/ P1(短信)/ P2(IM)/ P3(邮件)
3. 避免噪音:合并相关告警,设置合理的阈值和持续时间
4. 值班机制:轮值On-Call,明确响应时间
5. 告警收敛:抑制、静默、聚合
groups:
- name: java-app
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
> 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "错误率超过1%"
description: "{{ $labels.application }} 错误率 {{ $value | humanizePercentage }}"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
rate(http_server_requests_seconds_bucket[5m])
) > 3
for: 3m
labels:
severity: warning
annotations:
summary: "P99延迟超过3秒"
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default'
routes:
- match:
severity: critical
receiver: 'oncall-phone'
repeat_interval: 1h
- match:
severity: warning
receiver: 'team-chat'
inhibit_rules:
- source_match:
severity: critical
target_match:
severity: warning
equal: ['alertname', 'cluster']
轮值:每人1周,7×24小时
响应时间:P0 < 5分钟,P1 < 15分钟
升级:15分钟未响应 → 升级到下一级
复盘:每次P0/P1事后复盘(Post-Mortem)
工具:PagerDuty、OpsGenie、自建值班系统。
Spring Boot 3.x使用Micrometer Observation API统一了Tracing和Metrics:
Application Code
↓
Observation API (Micrometer)
↓
┌────────────────┬──────────────────┐
Metrics Handler Tracing Handler Log Handler
↓ ↓ ↓
Prometheus OTel/Zipkin Loki/ELK
↓ ↓ ↓
Grafana Dashboard Jaeger UI Kibana
# application.yml
management:
endpoints:
web:
exposure:
include: prometheus,health,info,metrics
metrics:
export:
prometheus:
enabled: true
tags:
application: ${spring.application.name}
distribution:
percentiles-histogram:
http.server.requests: true
percentiles:
http.server.requests: 0.5, 0.95, 0.99
自动采集的指标:
http.server.requests:HTTP请求延迟和计数jvm.memory.used/max:JVM内存jvm.gc.pause:GC暂停时间hikaricp.connections.active:连接池disk.total/free:磁盘
@Component
public class OrderMetrics {
private final Counter orderCounter;
private final Timer orderTimer;
private final Gauge pendingOrdersGauge;
public OrderMetrics(MeterRegistry registry, OrderRepository repo) {
orderCounter = Counter.builder("orders.created.total")
.description("Total orders created")
.tag("type", "normal")
.register(registry);
orderTimer = Timer.builder("orders.processing.duration")
.description("Order processing time")
.register(registry);
pendingOrdersGauge = Gauge.builder("orders.pending", repo::countPending)
.description("Pending orders count")
.register(registry);
}
public void recordOrder(Duration duration) {
orderCounter.increment();
orderTimer.record(duration);
}
}
<!-- 依赖 -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
# application.yml
management:
tracing:
sampling:
probability: 0.1 # 10%采样
otlp:
tracing:
endpoint: http://otel-collector:4317
自动采集:
应用层:Spring Boot Actuator + Micrometer
↓
采集层:Prometheus(Metrics)+ OTel Collector(Traces)+ Filebeat(Logs)
↓
存储层:Prometheus TSDB + Ja ES + Loki
↓
展示层:Grafana Dashboard(统一入口)
↓
告警层:Alertmanager + PagerDuty
服务Dashboard模板:
第一行:Golden Signals
第二行:JVM
第三行:依赖
第四行:业务
# Consumer Lag
kafka_consumer_group_lag
# 生产者发送速率
rate(kafka_producer_record_send_total[5m])
# 消费者消费速率
rate(kafka_consumer_records_consumed_total[5m])
# Broker磁盘使用
kafka_log_partition_size_bytes
# 副本ISR
kafka_cluster_partition_underreplicated
# 连接池使用率
hikaricp_connections_active / hikaricp_connections_max
# 慢查询(需要MySQL Exporter)
mysql_slow_queries_total
# 连接数
mysql_global_status_threads_connected
# 主从延迟
mysql_slave_status_seconds_behind_master
cAdvisor:容器指标(CPU/内存/网络/磁盘)
kube-state-metrics:K8s对象状态(Pod/Deployment/Node)
Node Exporter:节点系统指标
# Pod重启次数
kube_pod_container_status_restarts_total
# Pod非Ready
kube_pod_status_phase{phase!="Running"}
# CPU Request使用率
sum(rate(container_cpu_usage_seconds_total{container!="POD"}[5m]))
/
sum(kube_pod_container_resource_requests{resource="cpu"})
# 内存使用
container_memory_working_set_bytes{container!="POD"}
排查流程:
1. 告警触发 → Grafana查看大盘,确认影响范围
2. Golden Signals → 定位是延迟?错误?还是流量突增?
3. 链路追踪 → 找到慢的/失败的具体调用
4. 日志分析 → 查看错误日志的具体信息
5. 定位根因 → 代码Bug?资源不足?依赖故障?
6. 修复验证 → 部署修复后监控确认
1. 阈值设置不合理:P99 > 500ms但平时就是400ms
2. 缺少持续时间:单次超限就告警,应该是持续N分钟
3. 缺少基线:不同时段的正常值不同
4. 告警风暴:一个故障触发几十条告警
5. 采样率太低:1分钟采样可能漏掉瞬时问题
# 磁盘使用趋势(7天预测)
predict_linear(disk_usage_bytes[7d], 7*24*3600)
# 连接池趋势
predict_linear(hikaricp_connections_active[3d], 3*24*3600)
# QPS增长率
rate(http_server_requests_seconds_count[7d])
容量报告:每周自动生成,包含:
混沌实验:
1. 注入故障(网络延迟、Pod Kill、CPU压力)
2. 观察监控告警是否正常触发
3. 验证自愈机制是否生效
4. 记录MTTD(平均发现时间)和MTTR(平均恢复时间)
工具:Chaos Mesh、LitmusChaos、Gremlin。
Error Budget = 1 - SLO
SLO = 99.9%
Error Budget = 0.1%(每月约43分钟不可用)
已消耗 = 1 - 实际可用率
剩余Budget = Error Budget - 已消耗
用途:
第一步:基础设施监控(CPU/内存/磁盘/网络)
第二步:应用监控(QPS/延迟/错误率)
第三步:中间件监控(DB/Redis/Kafka)
第四步:业务监控(订单量/转化率)
第五步:链路追踪(分布式调用链)
第六步:告警体系(分级/值班/升级)
第七步:On-Call机制和Post-Mortem文化
| 维度 | Pull(Prometheus) | Push(Datadog/Cat) |
|---|---|---|
| 优点 | 简单、服务发现、健康检查 | 实时、不怕NAT/防火墙 |
| 缺点 | 不适合短生命周期任务 | 服务端压力大 |
| 适用 | 长期运行的服务 | Serverless、批处理任务 |
Spring Boot:默认Pull模式(Prometheus scrape /actuator/prometheus)
1. 减少指标基数(Cardinality):避免高基数标签(如user_id)
2. 合理设置采集间隔:15s vs 30s vs 60s
3. 使用Recording Rules:预计算复杂查询
4. 远程存储:VictoriaMetrics/Thanos长期存储
5. 分片:按服务分片Prometheus
# Recording Rules
groups:
- name: api
interval: 30s
rules:
- record: api:http_requests:rate5m
expr: sum(rate(http_server_requests_seconds_count[5m])) by (application, method, uri)
Cluster-1: Prometheus → Thanos Sidecar →─┐
Cluster-2: Prometheus → Thanos Sidecar →─┼→ Thanos Query → Grafana
Cluster-3: Prometheus → Thanos Sidecar →─┘ ↑
Thanos Store Gateway
↑
Object Storage(S3/MinIO)
AI特定指标:
// Spring AI Observability(自动采集)
// - ai.chat.token.usage:Token使用量
// - ai.chat.latency:LLM调用延迟
// - ai.chat.errors:调用失败数
// 自定义业务指标
Counter.builder("ai.rag.retrieval.count")
.tag("source", "milvus")
.register(registry).increment();
策略:
1. 分组(Group By):相同告警名的合并
2. 抑制(Inhibit):高优先级抑制低优先级
3. 静默(Silence):维护窗口期静默
4. 路由(Route):不同告警发不同团队
5. 去重(Dedup):相同告警不重复发送
"我负责了团队的可观测性体系建设。使用Prometheus + Grafana搭建了监控平台,包含4层Dashboard(基础设施/应用/中间件/业务)。引入OpenTelemetry实现了Metrics+Traces+Logs的关联查询。设计了分级告警体系(P0电话→P1短信→P2飞书),告警准确率从60%提升到95%。建立了On-Call值班制度和Post-Mortem复盘机制。"
# Grafana Terraform Provider
resource "grafana_dashboard" "service" {
config_json = jsonencode({
title = "Service Dashboard"
uid = "service-${var.service_name}"
panels = [
{
title = "QPS"
type = "timeseries"
targets = [{
expr = "rate(http_server_requests_seconds_count{application=\"${var.service_name}\"}[5m])"
}]
}
]
})
}
工具:Terraform、Grafonnet(Jsonnet)、Grafana Ansible。
自动扩缩容:
# K8s HPA + Prometheus
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
metrics:
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "1000"
自动重启:健康检查失败 → K8s自动重启Pod
自动降级:错误率 > 5% → 自动切换到备用服务
1. OpenTelemetry一统天下:Metrics+Traces+Logs统一采集
2. eBPF无侵入监控:内核级可观测性
3. AI辅助根因分析:自动关联异常和根因
4. Continuous Profiling:持续性能分析(Pyroscope/Parca)
5. Real User Monitoring(RUM):前端体验监控
6. GitOps管理Dashboard:Dashboard as Code
7. 成本可观测性:FinOps与监控结合
可观测三支柱:Metrics看趋势、Tracing查链路、Logs看细节
>
Golden Signals四指标:延迟、流量、错误、饱和度
>
告警三原则:可操作、分级、收敛
>
SRE三件套:SLI量化→SLO承诺→Error Budget平衡