本文是 Audio LLM 流式 ASR 与语音大模型工程化深度实战 的分章阅读版。
核心问题:1 张 A10 GPU 24GB,如何支撑 100 路并发 ASR?
关键指标:
模式 A:每会话独占模型(简单,低效)
# 每个用户一个模型实例,显存爆
for user in users:
model = whisper.load_model("large") # ❌ 不可能 100 个
模式 B:模型共享,batch 推理
# 单模型实例,batch 收集
class BatchAsrEngine:
def __init__(self):
self.model = whisper.load_model("large")
self.batch_queue = asyncio.Queue()
async def submit(self, audio_chunk):
await self.batch_queue.put(audio_chunk)
async def run(self):
while True:
# 收集 batch(max 32 个,等待 50ms)
batch = await self.collect_batch(max_size=32, timeout_ms=50)
# 一次 forward
results = self.model.batch_transcribe(batch)
# 分发结果
for r, callback in zip(results, batch.callbacks):
callback(r)
模式 C:Triton Inference Server(生产首选)
Triton 优势:
- 动态 batching(自动 batch,无需手动管理)
- 多模型实例(GPU 切片)
- 显存池化
- 监控指标(Prometheus)
部署:
- 模型 → ONNX / TensorRT
- config.pbtxt 配置 max_batch_size=32, dynamic_batching
- Java 端用 Triton gRPC 客户端
| 方案 | 加速比 | 精度损失 | 实施难度 |
|---|---|---|---|
| faster-whisper (CTranslate2) | 4x | 几乎无 | 低 |
| whisper.cpp (INT8) | 8x | 1-2% WER | 中 |
| Whisper-TensorRT | 10x | 1-2% | 高 |
| Whisper-Flash-Attention | 2x | 无 | 中 |
| distilled (Whisper-tiny) | 20x | 5% WER | 低 |
推荐组合:faster-whisper + INT8 + Flash-Attention,4-8 倍加速,精度损失 < 2%。
@Configuration
public class AsrClientConfig {
@Bean
public AsrClientPool asrClientPool(
@Value("${asr.triton.endpoints}") List<String> endpoints) {
// 多 Triton 节点负载均衡
return AsrClientPool.builder()
.endpoints(endpoints) // ["grpc://gpu1:8001", "grpc://gpu2:8001"]
.maxConnectionsPerEndpoint(50)
.connectTimeout(Duration.ofSeconds(5))
.readTimeout(Duration.ofSeconds(30))
.loadBalancer(LoadBalancerType.LEAST_CONNECTIONS)
.retryPolicy(RetryPolicy.builder().maxAttempts(3).build())
.circuitBreaker(CircuitBreaker.builder()
.failureRateThreshold(0.5f)
.waitDurationInOpenState(Duration.ofSeconds(30))
.build())
.build();
}
}
# Kubernetes HPA 配置
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: asr-engine
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: asr-engine
minReplicas: 3
maxReplicas: 30
metrics:
- type: Pods
pods:
metric:
name: asr_active_streams
target:
type: AverageValue
averageValue: "20" # 每个 Pod 最大 20 路流
- type: Resource
resource:
name: nvidia_com_gpu_utilization
target:
type: Utilization
averageUtilization: 70