Flink CDC sinks to Hudi on Databricks based on ADLS.
当前的架构
FlinkCDC --> Kafka -->Delta on Databrciks–(raw,stg,cur_biz)–>MySQL
当前面临的问题
实时性不够
| 阶段 | realtime performance |
|---|---|
| database source -> FlinkCDC | ✅️ |
| FlinkCDC -> Kafka | ✅️ |
| kafka -> raw | ❌️ |
| raw->stg | ❓️ |
| stg->curz_biz | ✅️ |
| cur_biz->MySQL | ❓️ |
新架构设计
方案一
FlinkCDC -->HuDi on Databrciks–(raw,stg,cur_biz)–>MySQL
参考文献
| Stage | reference links |
|---|---|
| FlinkCDC -->HuDi | https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/zh/docs |