yarn 的单作业运行模式
1.yarn会话模式的管理
#查看yarn应用的日志 yarn logs -applicationId application_1782739955886_0005 |less #查看的结果如下; Container: container_e02_1782739955886_0005_01_000002 on hadoop001_35005 LogAggregationType: AGGREGATED ======================================================================== LogType:directory.info LogLastModifiedTime:Mon Jun 29 22:37:40 +0800 2026 LogLength:6332 LogContents: ls -l: total 32 -rw-r--r-- 1 flink flink 7 Jun 29 22:37 container_tokens -rwx------ 1 flink flink 705 Jun 29 22:37 default_container_executor_session.sh -rwx------ 1 flink flink 760 Jun 29 22:37 default_container_executor.sh lrwxrwxrwx 1 flink flink 170 Jun 29 22:37 flink-conf.yaml -> /oracle/hadoop/tmp/nm-local-dir/usercache/flink/appcache/application_1782739955886_0005/filecache/24/application_1782739955886_0005-flink-conf.yaml7389539160443170593.tmp lrwxrwxrwx 1 flink flink 122 Jun 29 22:37 flink-dist-1.17.0.jar -> /oracle/hadoop/tmp/nm-local-dir/usercache/flink/appcache/application_1782739955886_0005/filecache/10/flink-dist-1.17.0.jar lrwxrwxrwx 1 flink flink 136 Jun 29 22:37 FlinkTuTorial-1.17-1.0-SNAPSHOT.jar -> /oracle/hadoop/tmp/nm-local-dir/usercache/flink/appcache/application_1782739955886_0005/filecache/15/FlinkTuTorial-1.17-1.0-SNAPSHOT.jar -rwx------ 1 flink flink 12658 Jun 29 22:37 launch_container.sh drwxrwxr-x 2 flink flink 4096 Jun 29 22:37 lib lrwxrwxrwx 1 flink flink 117 Jun 29 22:37 log4j.properties -> /oracle/hadoop/tmp/nm-local-dir/usercache/flink/appcache/application_1782739955886_0005/filecache/23/log4j.properties drwxrwxr-x 10 flink flink 210 Jun 29 22:37 plugins drwx--x--- 2 flink flink 6 Jun 29 22:37 tmp find -L . -maxdepth 5 -ls: 68675569 4 drwx--x--- 5 flink flink 4096 Jun 29 22:37 . 135775831 0 drwx--x--- 2 flink flink 6 Jun 29 22:37 ./tmp 68675570 4 -rw-r--r-- 1 flink flink 7 Jun 29 22:37 ./container_tokens 68675571 4 -rw-r--r-- 1 flink flink 12 Jun 29 22:37 ./.container_tokens.crc 68675572 16 -rwx------ 1 flink flink 12658 Jun 29 22:37 ./launch_container.sh 68675573 4 -rw-r--r-- 1 flink flink 108 Jun 29 22:37 ./.launch_container.sh.crc 68675574 4 -rwx------ 1 flink flink 705 Jun 29 22:37 ./default_container_executor_session.sh 68675575 4 -rw-r--r-- 1 flink flink 16 Jun 29 22:37 ./.default_container_executor_session.sh.crc 68675576 4 -rwx------ 1 flink flink 760 Jun 29 22:37 ./default_container_executor.sh#通过yarn-session命令行的方式也可以停止yarn应用。
echo "stop "| yarn-session.sh -id application_1782739955886_0005
#这个命令专门用于停止 Flink YARN Session 模式下的集群
2.yarn 的单作业运行模式
在 YARN 环境中,由于有了外部平台做资源调度,所以我们也可以直接向YARN 提交一个单独的作业,
从而启动一个 FLINK 集群。在客户端解析代码。
[flink@hadoop001 flink-1.17.0]$ bin/flink run -d -t yarn-per-job -c com.ycl.WordCountStreamUnboundedDemo lib/FlinkTuTorial-1.17-1.0-SNAPSHOT.jar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2026-06-29 22:36:01,965 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. 2026-06-29 22:36:01,965 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/module/flink-1.17.0/lib/flink-dist-1.17.0.jar) to field java.lang.String.value WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2026-06-29 22:36:02,580 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.17.0/conf')already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2026-06-29 22:36:02,823 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of classorg.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2026-06-29 22:36:02,836 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Job Clusters are deprecated since Flink 1.15. Please use an Application Cluster/Application Mode instead. 2026-06-29 22:36:02,907 INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to yarnrm2 2026-06-29 22:36:02,990 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found 2026-06-29 22:36:02,990 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'. 2026-06-29 22:36:03,042 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink. 2026-06-29 22:36:03,042 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MBmay not be used by Flink. 2026-06-29 22:36:03,042 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=1} 2026-06-29 22:36:11,759 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cannot use kerberos delegation token manager, no valid kerberos credentials provided. 2026-06-29 22:36:11,764 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1782739955886_0005 2026-06-29 22:36:12,021 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1782739955886_0005 2026-06-29 22:36:12,022 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated 2026-06-29 22:36:12,025 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED 2026-06-29 22:37:12,066 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster 2026-06-29 22:37:27,477 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully. 2026-06-29 22:37:27,478 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The Flink YARN session cluster has been started in detached mo de. In order to stop Flink gracefully, use the following command: $ echo "stop" | ./bin/yarn-session.sh -id application_1782739955886_0005 If this should not be possible, then you can also kill Flink via YARN's web interface or via: $ yarn application -kill application_1782739955886_0005 Note that killing Flink might not clean up all job artifacts and temporary files. 2026-06-29 22:37:27,479 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop003:15478 of application 'application_1782739955886_0005'. Job has been submitted with JobID 7ba5e9a55e7854651d57f43e85b7d280 Exception in thread "Thread-4" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'. at org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184) at org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:208) at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2839) at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3113) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3072) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3045) at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2923) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2905) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1247) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1864) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1841) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183) at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145) at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)bin/flink run -d -t yarn-per-job -c com.ycl.WordCountStreamUnboundedDemo lib/FlinkTuTorial-1.17-1.0-SNAPSHOT.jar
这个方式启动后,客户端退出,不会影响正在执行的作业。
#分离模式启动;客户端启动后在后台运行,不会一直在前台界面停留。
bin/flink run -d -t yarn-per-job -d -c com.ycl.WordCountStreamUnboundedDemo lib/FlinkTuTorial-1.17-1.0-SNAPSHOT.jar
查看或者取消作业;
#查看作业
bin/flink list -t yarn-per-job -Dyarn.application.id=application_1782910361073_0001
#取消作业
bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1782910361073_0001 <jobId>
application_1782910361073_0001 是当前应用的ID,<JOBID> 是作业的ID,如果取消作业,整个flink集群也会停掉。
单作业模式,作业停止后,集群关闭了,同时作业状态变成:Finished; 资源会被回收。
#在分离模式下的一个问题,从flink1.13版本就有。
you can disable this check with the configuration 'classloader.check-leaked-classloader'.
#可以通过配置关闭这个报错。
vim flink-conf.yaml
classloader.check-leaked-classloader: false
再次运行作业,看是否报错。 可以看到已经没有报错了。 [flink@hadoop001 flink-1.17.0]$ bin/flink run -d -t yarn-per-job -c com.ycl.WordCountStreamUnboundedDemo lib/FlinkTuTorial-1.17-1.0-SNAPSHOT.jar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2026-07-01 21:06:17,392 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. 2026-07-01 21:06:17,392 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/module/flink-1.17.0/lib/flink-dist-1.17.0.jar) to field java.lang.String.value WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2026-07-01 21:06:17,812 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.17.0/conf')already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2026-07-01 21:06:18,073 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of classorg.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2026-07-01 21:06:18,095 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Job Clusters are deprecated since Flink 1.15. Please use an Application Cluster/Application Mode instead. 2026-07-01 21:06:18,174 INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to yarnrm2 2026-07-01 21:06:18,250 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found 2026-07-01 21:06:18,250 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'. 2026-07-01 21:06:18,332 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink. 2026-07-01 21:06:18,332 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MBmay not be used by Flink. 2026-07-01 21:06:18,332 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=1} 2026-07-01 21:06:25,891 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cannot use kerberos delegation token manager, no valid kerberos credentials provided. 2026-07-01 21:06:25,896 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1782910361073_0002 2026-07-01 21:06:26,146 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1782910361073_0002 2026-07-01 21:06:26,146 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated 2026-07-01 21:06:26,150 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED 2026-07-01 21:06:32,916 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully. 2026-07-01 21:06:32,917 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The Flink YARN session cluster has been started in detached mode. In order to stop Flink gracefully, use the following command: $ echo "stop" | ./bin/yarn-session.sh -id application_1782910361073_0002 If this should not be possible, then you can also kill Flink via YARN's web interface or via: $ yarn application -kill application_1782910361073_0002 Note that killing Flink might not clean up all job artifacts and temporary files. 2026-07-01 21:06:32,917 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop001:30895 of application 'application_1782910361073_0002'. Job has been submitted with JobID 88f7bb56d52dc5493de32c90b0498ea0 #查看作业 bin/flink list -t yarn-per-job -Dyarn.application.id=application_1782910361073_0002#日志页面出现一个报错。
Failed while trying to construct the redirect url to the log server. Log Server url may not be configured
Local Logs:
java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.
#解决方法;
未配置 jobhistory 的访问路径。
vim yarn-site.xml
<property>
<name>yarn.log.server.url</name>
<value>http://hadoop001:19888/jobhistory/logs</value>
</property>
#查看作业 bin/flink list -t yarn-per-job -Dyarn.application.id=application_1782912326958_0004 [flink@hadoop001 flink-1.17.0]$ bin/flink list -t yarn-per-job -Dyarn.application.id=application_1782912326958_0004 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2026-07-01 21:35:25,159 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. 2026-07-01 21:35:25,159 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. 2026-07-01 21:35:25,315 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.17.0/conf')already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2026-07-01 21:35:25,527 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of classorg.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2026-07-01 21:35:25,580 INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to yarnrm2 2026-07-01 21:35:25,626 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop002:18524 of application 'application_1782912326958_0004'. Waiting for response... ------------------ Running/Restarting Jobs ------------------- 01.07.2026 21:34:58 : 1c60c35310ebcfe2a4601f40c1d6a0b4 : Flink Streaming Job (RUNNING) -------------------------------------------------------------- No scheduled jobs.socke 7777 端口写入数据,flink消费
#取消作业 -t:指定模式,-D指定应用名称; bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1782912326958_0004 1c60c35310ebcfe2a4601f40c1d6a0b4 [flink@hadoop001 flink-1.17.0]$ bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_1782912326958_0004 1c60c35310ebcfe2a4601f40c1d6a0b4 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/flink-1.17.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2026-07-01 21:43:18,593 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. 2026-07-01 21:43:18,593 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink. Cancelling job 1c60c35310ebcfe2a4601f40c1d6a0b4. 2026-07-01 21:43:18,761 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/opt/module/flink-1.17.0/conf')already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2026-07-01 21:43:19,039 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of classorg.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2026-07-01 21:43:19,103 INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to yarnrm2 2026-07-01 21:43:19,172 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop002:18524 of application 'application_1782912326958_0004'. Cancelled job 1c60c35310ebcfe2a4601f40c1d6a0b4.