共50道题 | 难度分布:★★★★☆平均 | 20K+岗位大数据方向加分项
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★☆☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★★
答案要点:
难度:★★★★☆
答案要点:
day(timestamp)→按天分区bucket(16, user_id)→16个桶truncate(10, name)→截取前10字符难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
SELECT * FROM table Increments
WHERE snapshot_id BETWEEN start_snapshot AND end_snapshot
_change_type列标识(I/U/D)IncrementalAppendScan/IncrementalChangelogScan难度:★★★★☆
答案要点:
_delta_log/目录存储JSON+Parquet格式的日志delta-spark或delta-standalone(无Spark依赖)难度:★★★★★
答案要点:
_delta_log/目录:存储所有事务操作记录难度:★★★★☆
答案要点:
MERGE INTO target USING source ON target.id = source.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
难度:★★★★☆
答案要点:
OPTIMIZE table ZORDER BY (user_id, date)难度:★★★★☆
答案要点:
io.delta:delta-spark_2.12:3.xspark.sql.extensions=io.delta.sql.DeltaSparkSessionExtensionspark.read.format("delta").load("/path/to/table")df.write.format("delta").mode("append").save("/path")CREATE TABLE USING DELTA LOCATION '/path'spark.readStream.format("delta")/writeStream.format("delta")DeltaLog/Snapshot/DeltaTable类难度:★★★★☆
答案要点:
难度:★★★★★
答案要点:
难度:★★★★☆
答案要点:
cleaner.policy=KEEP_LATEST_COMMITS保留最近N个版本难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
compaction.trigger.strategy=NUM_COMMITS:每N次提交触发compaction.trigger.strategy=TIME_ELAPSED:每N小时触发compaction.trigger.strategy=NUM_AND_TIME:满足任一条件触发compaction.delta_commits(默认5)控制触发频率难度:★★★★★
答案要点:
| 维度 | Delta Lake | Iceberg | Hudi |
|---|---|---|---|
| 开源方 | Databricks | Netflix/Apple | Uber |
| ACID | 支持 | 支持 | 支持 |
| Upsert | MERGE | MOR表 | 原生支持(最强) |
| Schema演进 | 支持 | 最灵活 | 支持 |
| 分区演进 | 有限 | 最强 | 有限 |
| 多引擎 | Spark为主 | 所有引擎 | Spark/Flink |
| 增量读取 | Change Data Feed | Incremental Scan | 原生支持 |
| 社区活跃度 | 高(Databricks) | 最高 | 高 |
| 学习曲线 | 低 | 中 | 高 |
难度:★★★★★
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★★
答案要点:
VACUUM(Delta)/clean(Hudi)/expire_snapshots(Iceberg)难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
IcebergUtil类直接操作Iceberg表难度:★★★★★
答案要点:
CREATE TABLE mysql_source WITH ('connector'='mysql-cdc', ...);
CREATE TABLE iceberg_sink WITH ('connector'='iceberg', ...);
INSERT INTO iceberg_sink SELECT * FROM mysql_source;
难度:★★★★☆
答案要点:
spring.datasource.url=jdbc:trino://trino:8080/iceberg/data_lakespark-submit --class ThriftServer启动JDBC服务难度:★★★★★
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
SELECT * FROM table VERSION AS OF 123难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★★
答案要点:
难度:★★★★★
答案要点:
难度:★★★★★
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★☆
答案要点:
难度:★★★★★
答案要点:
难度:★★★☆☆
答案要点: