20K Java+AI 岗位高频考点 — Elasticsearch是全文检索和日志分析的标准方案,面试常考倒排索引、分词、聚合、Spring Data ES集成。覆盖ES架构、索引设计、查询DSL、性能优化、Java集成。
ES = 分布式搜索和分析引擎(基于Lucene)
概念对照:
MySQL → Elasticsearch
Database → Index
Table → Type(7.x后废弃,一个Index一个Type)
Row → Document(JSON)
Column → Field
Index → Inverted Index(倒排索引)
核心特性:
- 全文检索(倒排索引)
- 近实时搜索(1秒延迟)
- 分布式(自动分片和副本)
- RESTful API
- 聚合分析
- 多种数据类型(text/keyword/integer/date/nested)
版本:
7.x:去掉Type
8.x:安全默认开启,REST Client推荐
正排索引:文档ID → 内容
倒排索引:关键词 → 文档ID列表
示例:
文档1:Java编程指南
文档2:Python编程入门
文档3:Java并发编程
倒排索引:
Term → DocIDs
Java → [1, 3]
编程 → [1, 2, 3]
指南 → [1]
Python → [2]
入门 → [2]
并发 → [3]
查询"Java编程":
1. 分词:[Java, 编程]
2. 查倒排索引:
Java → [1, 3]
编程 → [1, 2, 3]
3. 交集:[1, 3]
4. 相关性评分(TF-IDF/BM25)排序
分词器(Analyzer):
Character Filter → Tokenizer → Token Filter
HTML去除 分词 大小写/停用词/词干
中文分词:
- ik_max_word:最细粒度切分
- ik_smart:智能切分(推荐索引时max_word,搜索时smart)
节点类型:
- Master Node:集群管理(元数据、索引创建、分片分配)
- Data Node:存储数据和执行查询
- Coordinating Node:请求路由和结果合并
- Ingest Node:数据预处理(Pipeline)
分片(Shard):
- Primary Shard:主分片(写入和读取)
- Replica Shard:副本分片(读取和备份)
- 分片数在索引创建后不能修改
- 每个分片是一个Lucene索引
写入流程:
Client → Coordinating Node → 定位Primary Shard →
Primary写入 → 同步到Replica → 返回成功
查询流程:
Client → Coordinating Node → 广播到所有Shard →
各Shard返回Top-N → Coordinating合并排序 → 返回
关键参数:
cluster.name: my-cluster
node.roles: [data, master]
index.number_of_shards: 3
index.number_of_replicas: 1
PUT /products
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"ik_smart_analyzer": {
"type": "custom",
"tokenizer": "ik_smart"
}
}
}
},
"mappings": {
"properties": {
"name": {
"type": "text",
"analyzer": "ik_max_word",
"search_analyzer": "ik_smart",
"fields": {
"keyword": { "type": "keyword" }
}
},
"category": { "type": "keyword" },
"price": { "type": "scaled_float", "scaling_factor": 100 },
"sales": { "type": "integer" },
"description": {
"type": "text",
"analyzer": "ik_max_word"
},
"tags": { "type": "keyword" },
"created_at": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" },
"specs": { "type": "nested",
"properties": {
"name": { "type": "keyword" },
"value": { "type": "keyword" }
}
}
}
}
}
设计原则:
1. text vs keyword:需要全文检索用text,精确匹配用keyword
2. multi-field:text+keyword共存(name.text / name.keyword)
3. nested:数组对象独立索引(避免扁平化问题)
4. date:统一时间格式
5. scaled_float:金额用缩放浮点
PUT _index_template/logs
{
"index_patterns": ["app-logs-*"],
"priority": 100,
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "logs-policy",
"index.lifecycle.rollover_alias": "app-logs"
},
"mappings": {
"properties": {
"timestamp": { "type": "date" },
"level": { "type": "keyword" },
"service": { "type": "keyword" },
"message": { "type": "text", "analyzer": "standard" },
"traceId": { "type": "keyword" }
}
}
}
}
// 按日期自动滚动索引
// app-logs-2024.01.01, app-logs-2024.01.02...
// 配合ILM(Index Lifecycle Management)自动删除旧索引
// 1. 全文检索
GET /products/_search
{
"query": {
"multi_match": {
"query": "Java编程",
"fields": ["name^3", "description"],
"type": "best_fields"
}
}
}
// 2. 精确匹配
GET /products/_search
{
"query": {
"bool": {
"must": [
{ "term": { "category": "书籍" } },
{ "range": { "price": { "gte": 50, "lte": 200 } } }
],
"should": [
{ "term": { "tags": "热销" } }
],
"must_not": [
{ "term": { "status": "下架" } }
]
}
}
}
// 3. 聚合分析
GET /products/_search
{
"size": 0,
"aggs": {
"by_category": {
"terms": { "field": "category", "size": 10 },
"aggs": {
"avg_price": { "avg": { "field": "price" } },
"total_sales": { "sum": { "field": "sales" } }
}
},
"price_ranges": {
"range": {
"field": "price",
"ranges": [
{ "key": "低价", "to": 50 },
{ "key": "中价", "from": 50, "to": 200 },
{ "key": "高价", "from": 200 }
]
}
}
}
}
// 4. 高亮
GET /products/_search
{
"query": { "match": { "name": "Java" } },
"highlight": {
"fields": {
"name": { "pre_tags": ["<em>"], "post_tags": ["</em>"] }
}
}
}
// 文档定义
@Document(indexName = "products")
public class Product {
@Id private String id;
@Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_smart")
private String name;
@Field(type = FieldType.Keyword) private String category;
@Field(type = FieldType.Double) private Double price;
@Field(type = FieldType.Integer) private Integer sales;
}
// Repository
public interface ProductRepository extends ElasticsearchRepository<Product, String> {
List<Product> findByCategory(String category);
Page<Product> findByName(String name, Pageable pageable);
}
// 复杂查询(ElasticsearchOperations)
@Service
public class ProductSearchService {
@Autowired ElasticsearchOperations esOps;
public SearchHits<Product> search(String keyword, String category,
Double minPrice, Double maxPrice,
int page, int size) {
BoolQueryBuilder bool = QueryBuilders.boolQuery();
if (keyword != null) {
bool.must(QueryBuilders.multiMatchQuery(keyword, "name", "description"));
}
if (category != null) {
bool.filter(QueryBuilders.termQuery("category", category));
}
if (minPrice != null || maxPrice != null) {
var range = QueryBuilders.rangeQuery("price");
if (minPrice != null) range.gte(minPrice);
if (maxPrice != null) range.lte(maxPrice);
bool.filter(range);
}
NativeQuery query = NativeQuery.builder()
.withQuery(bool)
.withPageable(PageRequest.of(page, size))
.withHighlightBuilder(new HighlightBuilder().field("name"))
.withSort(SortBuilders.fieldSort("sales").order(SortOrder.DESC))
.build();
return esOps.search(query, Product.class);
}
}
写入优化:
1. 批量写入(Bulk API,1000条/批)
2. 增大refresh_interval(默认1s → 30s)
写入密集期设置 index.refresh_interval = -1(关闭自动刷新)
3. 增大translog flush间隔
4. 关闭副本(写入完成后再开启)
5. 使用自动生成的ID(避免版本检查)
查询优化:
1. 使用filter代替query(filter有缓存,不计分)
2. 避免深分页(from+size超过10000)
→ 用search_after或scroll
3. 只返回需要的字段(_source过滤)
4. 预索引(把计算提前到索引时)
5. 使用routing减少分片扫描
分片设计:
1. 分片数 = 数据量 / 50GB(每个分片不超过50GB)
2. 分片数 = 节点数 × (1~3)
3. 副本数:生产至少1个
4. 热温冷架构(ILM管理)
JVM调优:
- 堆内存 = 物理内存的50%(不超过32GB)
- -Xms = -Xmx
- 使用G1GC
方案一:同步双写(简单但不推荐)
// 写MySQL同时写ES
orderMapper.insert(order);
productRepository.save(toDocument(order));
// 问题:一致性差,性能差
方案二:异步消息(推荐)
MySQL → Canal监听Binlog → Kafka → Logstash/Consumer → ES
方案三:Canal直接同步
MySQL → Canal → ES
方案四:Logstash JDBC Input
定时查询MySQL增量数据 → 写入ES
适合对实时性要求不高的场景
推荐方案:Canal + Kafka + Consumer
- 实时性高(秒级延迟)
- 对业务代码无侵入
- Kafka削峰缓冲
| 维度 | Elasticsearch | Solr | Meilisearch |
|------|-------------|------|-------------|
| 语言 | Java | Java | Rust |
| 实时性 | 近实时 | 近实时 | 实时 |
| 分布式 | 原生 | 需配置 | 单机 |
| 聚合 | 强大 | 强大 | 有限 |
| 中文分词 | IK插件 | IK | 内置 |
| 运维 | 复杂 | 复杂 | 简单 |
| 适用 | 日志/搜索/分析 | 搜索引擎 | 小型搜索 |
选型:
- 日志分析(ELK) → Elasticsearch
- 电商搜索 → Elasticsearch
- 小型项目搜索 → Meilisearch/Typesense
ES核心:倒排索引(词→文档ID) + 分词器(Analyzer) + 相关性评分(BM25)
>
索引设计:text全文检索/keyword精确匹配/nested嵌套对象/multi-field多字段
>
查询三类型:全文(match/multi_match)、精确(term/range)、聚合(terms/avg/sum)
>
性能四优化:批量写入、filter缓存、避免深分页、合理分片