当前位置:

通过 docker-compose 部署 Flink

访客 2024-02-07 1167 0

概要

通过 docker-compose 以 Session Mode 部署 flink

前置依赖

  • Docker、docker-compose
  • flink 客户端
  • docker-compose.yml
version: "2.2"services:jobmanager:image: flink:1.17.2ports:- "8081:8081"command: jobmanagervolumes:- ${PWD}/checkpoint:/opt/flink/checkpoint- ${PWD}/savepoint:/opt/flink/savepointenvironment:- |FLINK_PROPERTIES=jobmanager.rpc.address: jobmanagertaskmanager:image: flink:1.17.2depends_on:- jobmanagercommand: taskmanagerscale: 1volumes:- ${PWD}/checkpoint:/opt/flink/checkpoint- ${PWD}/savepoint:/opt/flink/savepointenvironment:- |FLINK_PROPERTIES=jobmanager.rpc.address: jobmanagertaskmanager.numberOfTaskSlots: 3

部署

# 下载 docker、docker-compose# docker-compose 部署# 复制上述 docker-compose.yml 内容$ vim docker-compose.yml# 创建 savepoint、checkpoint 目录用于保存状态数据$ mkdir savepoint checkpoint$ docker-compose up -d

提交任务

# 下载 flink 客户端$ wget https://archive.apache.org/dist/flink/flink-1.17.0/flink-1.17.0-bin-scala_2.12.tgz$ tar xvf flink-1.17.0-bin-scala_2.12.tgz$ cd flink-1.17.0-bin-scala_2.12# 提交任务$ bin/flink run -d -m 192.168.56.103:8081 -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar# 查看任务列表$ bin/flink list -m 192.168.56.103:8081Waiting for response...------------------ Running/Restarting Jobs -------------------02.02.2024 16:45:01 : 9f70a7c1430d609eaa3434c4f30d1e93 : Flink-CDC (RUNNING)--------------------------------------------------------------No scheduled jobs

Web UI 查看

浏览器访问目标端口 192.168.56.103:8081

通过 docker-compose 部署 Flink

其它常用命令

  • 从指定 checkpoint 恢复任务
bin/flink run -d -m 192.168.56.103:8081 -s file:///opt/flink/checkpoint/47bea16a8ffcb51beff21ed1c7297dc1/chk-129773 -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar
  • 从指定 savepoint 恢复任务
bin/flink run -d -m 192.168.56.103:8081 -s file:///opt/flink/savepoint/savepoint-63712d-d7cbfee1ede9  -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar
  • 保存 savepoint
bin/flink savepoint 9f70a7c1430d609eaa3434c4f30d1e93 file:///opt/flink/savepoint -m 192.168.56.103:8081
  • 取消任务
bin/flink cancel 9f70a7c1430d609eaa3434c4f30d1e93 -m 192.168.56.103:8081
  • 停止任务并保存 savepoint
bin/flink stop 9f70a7c1430d609eaa3434c4f30d1e93 -p file:///opt/flink/savepoint -m 192.168.56.103:8081

参考

  • https://blog.csdn.net/qq_41538097/article/details/129113866 docker 搭建 flink 并上传任务
  • https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#flink-with-docker-compose Flink with Docker Compose
  • https://flink.apache.org/downloads/ Apache Flink® Downloads
  • https://blog.csdn.net/qq_26838315/article/details/125246223 Flink部署模式介绍(session,per-job,application)
  • https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#session-mode-1 Session Mode

发表评论

  • 评论列表
还没有人评论,快来抢沙发吧~